be-gay-do-crimes
/
NewPipeExtractor
mirror of https://github.com/TeamNewPipe/NewPipeExtractor
1
0
Fork 0
Code Issues Releases Wiki Activity
2,155 Commits 10 Branches 75 Tags 153 MiB
Commit Graph

2155 Commits

Author SHA1 Message Date
litetex 5bbea3a8f2 Fix ``YoutubeMixPlaylistExtractorTest`` again 
Doesn't work for kurgesagt videos anymore, used music video (copyright free) instead
 2022-08-14 15:12:27 +02:00
litetex da06166065 Updated mock data of ``YoutubeSearchExtractorTest$Suggestion`` 2022-08-14 15:12:26 +02:00
litetex 2e36ab1578 Fixed ``YoutubeSearchExtractorTest$Suggestion`` 2022-08-14 15:12:26 +02:00
litetex 2a8a623643 ``YoutubePlaylistLinkHandlerFactoryTest``: Use parameterized tests 2022-08-14 14:48:31 +02:00
litetex 938e69a16a Fixed ``YoutubePlaylistLinkHandlerFactoryTest`` 2022-08-14 14:48:30 +02:00
litetex 844de3e378 Fix ``YoutubeStreamLinkHandlerFactoryTest `` and parameterized the tests 
used code from #833
 2022-08-14 14:48:30 +02:00
litetex c1a72b8240 Use Android Studios code style 2022-08-14 14:48:29 +02:00
litetex fc27b8a5b8 Use preexisting ``ContentNotAvailableException`` 2022-08-14 14:48:29 +02:00
Stypox 17bad6cedf Fix test about exception type: NPE, not IllegalArgment 
The relevant change was made in #877
 2022-08-14 14:48:28 +02:00
litetex ecfc370685 Fixed all YTMixPlaylists 
Added option to choose if you want to consent or not - currently this is done by a static variable in ``YoutubeParsingHelper`` - may not be the best long-term solution but for now the tests work again (in EU countries) 🥳
 2022-08-14 14:48:27 +02:00
litetex 2b6fe294b2 Fixed ``YoutubeMusicSearchExtractorTest`` 
The renderers and queries got changed.
 2022-08-14 14:48:27 +02:00
litetex d6586da614 Failing test works locally without any problems. Reformatted it a bit. 2022-08-14 14:48:27 +02:00
litetex bf3ae5e679 Fixed ``SoundcloudStreamLinkHandlerFactoryTest`` 
* Removed the unnecessary ``public``
* Use parameterized tests
* One song got removed (which caused the test failure), replaced it with another song
 2022-08-14 14:48:27 +02:00
litetex 504f81036e Fixed ``PeertubePlaylistExtractorTest`` 
Also removed the unnecessary ``public`` and reformatted the code a bit
 2022-08-14 14:48:26 +02:00
litetex e7c12258f4 Fixed ``BandcampSearchExtractorTest`` 
Also removed the unnecessary ``public``
 2022-08-14 14:48:26 +02:00
litetex 9e8724df4d Fixed ``YoutubeStreamExtractorLivestreamTest`` 
The "Lofi Girl"-stream got interrupted by a copyright strike and had to be restarted. Because of this a new id is now used.
 2022-08-14 14:48:26 +02:00
Tobi 76aad92fa5
Merge pull request #890 from AudricV/yt-clients-update-improvements-and-extraction-fixes 
[YouTube] Update clients version and improve extraction of API key and client version of WEB client
 2022-08-12 23:32:41 +02:00
AudricV 472f5d9e9c
[YouTube] Update mocks 2022-08-12 19:20:32 +02:00
AudricV 7bdca33a87
[YouTube] Ensure that an additional player response is the correct one 
If YouTube detect that requests come from a third party client, they may
replace the real player response by another one of a video saying that this
content is not available on this app and to watch it on the latest version of
YouTube. We can detect this by checking whether the video ID of the player
response returned is the same as the one requested by the extractor.
 2022-08-12 19:20:31 +02:00
AudricV c82317e318
[YouTube] Spoof more mobile clients 
Additional parameters have been added to the player requests of ANDROID and IOS
clients:

- for both clients: osName and osVersion: their respective values are:
  - for the ANDROID one: Android and 12;
  - for the IOS one: iOS and 15.6.0.19G71.
- for the ANDROID client: androidTargetSdkVersion, with the Android SDK version
  corresponding to the Android version used in the player requests of this
  client. This parameter is now required with this client to be sure to get a
  correct player response, otherwise, the one of a video saying that this
  content is not available in this app and to watch it with the latest version
  of YouTube can be returned instead;
- for the IOS client: deviceMake, with Apple as its value.

The iOS version sent in the IOS client player requests has been also updated to
the version 15.6 of the OS.

Finally, a comment about the requirement to use the signature timestamp from
the player JavaScript base file for HTML5 player requests on videos with
obfuscated URLs has been added and replaces a previous one which may be not
true.
 2022-08-12 19:20:31 +02:00
AudricV d0549a5a52
[YouTube] Update client versions and use a real version for the iOS client 
The iOS version can be got easily in fact, by looking at the What's New section of the App Store' app page.
 2022-08-12 19:20:31 +02:00
AudricV d7e678aca2
[YouTube] Improve WEB client version and API key HTML extraction 
Common code in WEB client version HTML extraction has been deduplicated, usage of the Java 8 Stream API has been made and initial data fallback has been used as a last resort.
This means that the client version extraction from regexes will be used before this fallback, as it doesn't contain the full client version.
This can be used as a way to fingerprint the extractor, even if it seems to be not the case.
 2022-08-12 19:20:30 +02:00
AudricV 6a885ef5ab
Merge pull request #891 from Theta-Dev/fix-throttling-decrypter 
[YouTube] Fix extraction of more complex nsig functions
 2022-08-12 18:24:21 +02:00
AudricV 5b548340e8
[YouTube] Catch any exception in YoutubeThrottlingDecrypter.apply and improve docs 
This will prevent any future extractor break due to decryption failure, like it was excepted to be the case before.

Some documentation about the throttling decryption has been also improved.
 2022-08-12 17:49:36 +02:00
ThetaDev 52ded6e3d7
Handle curly braces inside strings in StringUtils.matchToClosingParenthesis 
This is required to extract fully more complex YouTube nsig functions.
 2022-08-12 16:32:00 +02:00
Stypox d12003651b
Merge pull request #886 from Isira-Seneviratne/toArray_improvements 
Make improvements to methods using toArray().
 2022-08-06 22:34:23 +02:00
Isira Seneviratne 7daca10a06 Make improvements to methods using toArray(). 2022-08-06 05:21:12 +05:30
Stypox 2906be22af
Merge pull request #881 from Isira-Seneviratne/String_join 
Use String.join() and Collectors.joining().
 2022-08-04 12:09:33 +02:00
Stypox 4ddb96a86f
Merge pull request #883 from TeamNewPipe/dependabot/gradle/com.google.code.gson-gson-2.9.1 
Bump gson from 2.9.0 to 2.9.1
 2022-08-04 11:20:41 +02:00
Isira Seneviratne 64771c5712 Use String.join() and Collectors.joining(). 2022-08-04 05:18:13 +05:30
Stypox fc8b5ebbc6
Merge pull request #878 from Isira-Seneviratne/Use_Collections 
Use Collections methods.
 2022-08-03 22:50:41 +02:00
Stypox 4a4939d89c
Merge pull request #877 from Isira-Seneviratne/Use_Objects_requireNonNull 
Use Objects.requireNonNull().
 2022-08-03 22:45:35 +02:00
Stypox c336bd58a5
Merge pull request #879 from TeamNewPipe/dependabot/gradle/org.junit-junit-bom-5.9.0 
Bump junit-bom from 5.8.2 to 5.9.0
 2022-08-03 21:06:20 +02:00
dependabot[bot] 325af31e5f
Bump gson from 2.9.0 to 2.9.1 
Bumps [gson](https://github.com/google/gson) from 2.9.0 to 2.9.1.
- [Release notes](https://github.com/google/gson/releases)
- [Changelog](https://github.com/google/gson/blob/master/CHANGELOG.md)
- [Commits](https://github.com/google/gson/compare/gson-parent-2.9.0...gson-parent-2.9.1)

---
updated-dependencies:
- dependency-name: com.google.code.gson:gson
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-08-01 09:26:14 +00:00
dependabot[bot] d905636021
Bump junit-bom from 5.8.2 to 5.9.0 
Bumps [junit-bom](https://github.com/junit-team/junit5) from 5.8.2 to 5.9.0.
- [Release notes](https://github.com/junit-team/junit5/releases)
- [Commits](https://github.com/junit-team/junit5/compare/r5.8.2...r5.9.0)

---
updated-dependencies:
- dependency-name: org.junit:junit-bom
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-07-27 09:20:12 +00:00
Isira Seneviratne 1af6b8eedb Use Collections.singletonList(). 2022-07-27 07:35:57 +05:30
Isira Seneviratne ff60e05c76 Use Collections.singletonMap(). 2022-07-27 07:35:57 +05:30
Isira Seneviratne 682a4263e5 Use Objects.requireNonNull(). 2022-07-27 06:55:26 +05:30
Stypox 8c5f014a6f
Merge pull request #869 from mhmdanas/add-workflow-permissions 
Use minimum required permissions for GitHub workflows
 2022-07-13 18:58:32 +02:00
Stypox 954a294e27
Merge pull request #870 from TeamNewPipe/dependabot/gradle/org.jsoup-jsoup-1.15.2 
Bump jsoup from 1.15.1 to 1.15.2
 2022-07-13 17:48:06 +02:00
Mohammed Anas f57049d2c0
Use temurin instead of adopt (#868) 
See
https://blog.adoptopenjdk.net/2021/08/goodbye-adoptopenjdk-hello-adoptium/.
 2022-07-10 18:38:08 +03:00
Stypox 5bd7bf20cc
Merge pull request #867 from TeamNewPipe/dependabot/gradle/com.github.spotbugs-spotbugs-annotations-4.7.1 
Bump spotbugs-annotations from 4.7.0 to 4.7.1
 2022-07-06 14:23:44 +02:00
Stypox 5ab74b3631
Merge pull request #857 from FireMasterK/video-title 
Get original untranslated title for YouTube
 2022-07-06 10:26:45 +02:00
dependabot[bot] 122365005a
Bump jsoup from 1.15.1 to 1.15.2 
Bumps [jsoup](https://github.com/jhy/jsoup) from 1.15.1 to 1.15.2.
- [Release notes](https://github.com/jhy/jsoup/releases)
- [Changelog](https://github.com/jhy/jsoup/blob/master/CHANGES)
- [Commits](https://github.com/jhy/jsoup/compare/jsoup-1.15.1...jsoup-1.15.2)

---
updated-dependencies:
- dependency-name: org.jsoup:jsoup
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-07-04 09:38:16 +00:00
mhmdanas 414186cff5 Use minimum required permissions for GitHub workflows 
This reduces the attack surface if the workflows are ever compromised.
 2022-07-03 23:55:28 +03:00
dependabot[bot] e9b4be3e3c
Bump spotbugs-annotations from 4.7.0 to 4.7.1 
Bumps [spotbugs-annotations](https://github.com/spotbugs/spotbugs) from 4.7.0 to 4.7.1.
- [Release notes](https://github.com/spotbugs/spotbugs/releases)
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.7.0...4.7.1)

---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-06-28 09:15:07 +00:00
Stypox 5219a705ba
Merge pull request #864 from AudricV/fetch-android-client-for-ended/post-livestreams 
[YouTube] Fetch the ANDROID client for ended/post livestreams
 2022-06-24 14:55:20 +02:00
AudricV 090debd83b
[YouTube] Fetch the ANDROID client for ended/post livestreams 
The ANDROID client was only fetched for video contents, where it can be useful on ended/post livestreams, if the n parameter of the WEB client cannot be decrypted, to avoid throttling issues (because the WEB client was only used before for ended/post livestreams).

It also provides an exclusive 48kbps M4A audio format in the adaptiveFormats array of the JSON player response, like other mobile clients (which can be also extracted from the response of the DASH manifest URL returned into the WEB client player's response, but the DASH manifest is not used by the extractor).

A note about non-fatality of fetching or parsing issues of the ANDROID and IOS clients has been added.
 2022-06-21 18:53:49 +02:00
litetex a26bcc55c4
Merge pull request #845 from TeamNewPipe/dependabot/gradle/com.github.spotbugs-spotbugs-annotations-4.7.0 
Bump spotbugs-annotations from 4.6.0 to 4.7.0
 2022-06-19 15:36:55 +02:00
dependabot[bot] 424eb1c559
Bump spotbugs-annotations from 4.6.0 to 4.7.0 
Bumps [spotbugs-annotations](https://github.com/spotbugs/spotbugs) from 4.6.0 to 4.7.0.
- [Release notes](https://github.com/spotbugs/spotbugs/releases)
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.6.0...4.7.0)

---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-06-19 13:29:18 +00:00