Commit Graph

2128 Commits

Author SHA1 Message Date
Stypox 5c83409039
[YouTube] Rewrite manifest test and rename long methods 2022-05-28 12:00:51 +02:00
Stypox 8226fd044f
[YouTube] Suppress Sonar security warning for XEE 2022-05-28 12:00:50 +02:00
Stypox ba68b8c014
[YouTube] Secure DashManifestCreator against XEE attack
Also remove duplicate lines from javadoc comments, otherwise checkstyle complaints.
XEE prevention is inspired from https://github.com/ChuckerTeam/chucker/pull/201/files
For an explanation of XEE see https://rules.sonarsource.com/java/RSPEC-2755, though the solution reported there causes crashes on Android
2022-05-28 12:00:50 +02:00
Stypox 159d05c91b
Remove unused DashMpdParser (but kept in git history) 2022-05-28 12:00:49 +02:00
Stypox 50272db946
Apply reviews: improve comments, remove FILE, remove Stream#equals(Stream) 2022-05-28 12:00:49 +02:00
TiA4f8R 07b045f20d
[YouTube] Support the iOS client in YoutubeDashManifestCreator and decrypt again the n parameter, if present, for all clients
This commits reverts a new behavior introduced in this branch, which only applied the decryption if needed on streams from the WEB client.
Also fix rebase issues and documentations style in YoutubeDashManifestCreator.
2022-05-28 12:00:48 +02:00
TiA4f8R 436ddde29f
Use assertThrows in YoutubeDashManifestCreatorTest 2022-05-28 12:00:48 +02:00
TiA4f8R d64d7bbd01
Move ManifestCreatorCache tests to a separate class and remove override of equals and hashCode methods in ManifestCreatorCache
These methods don't need to be overriden, as they are not excepted to be used in collections.
Also improve the toString method of this class, which contains also now clearFactor and maximumSize attributes and for each operations.
2022-05-28 12:00:48 +02:00
TiA4f8R 2fb1a412a6
Fix Checkstyle issues, revert resolution string changes for YouTube video streams and don't return the rn parameter in DASH manifests
This parameter is still used to get the initialization sequence of OTF and POST-live streams, but is not returned anymore in the manifests.
It has been removed in order to avoid fingerprinting based on the number sent (e.g. when starting to play a stream close to the end and using 123 as the request number where it should be 1) and should be added dynamically by clients in their requests.
The relevant test has been also updated.

Checkstyle issues in YoutubeDashManifestCreator have been fixed, and the changes in the resolution string returned for video streams in YoutubeStreamExtractor have been reverted, as they create issues on NewPipe right now.
2022-05-28 12:00:47 +02:00
TiA4f8R f61e2092a1
[YouTube] Return a copy of the hardcoded ItagItem instead of returning the reference to the hardcoded one in ItagItem.getItag
To do so, a copy constructor has been added in the class.

This fixes, for instance, an issue in NewPipe, in which the ItagItem values where not the ones corresponsing to a stream but to another, when generating DASH manifests.
2022-05-28 12:00:47 +02:00
TiA4f8R aa4c10e751
Improve documentation and adress most of the requested changes
Also fix some issues in several places, in the code and the documentation.
2022-05-28 12:00:46 +02:00
TiA4f8R 6985167e63
Add tests for YoutubeDashManifestCreator and ManifestCreatorCache
The test added in YoutubeDashManifestCreator uses a video of the Creative Commons channel, licenced under the Creative Commons Attribution licence (reuse allowed).
Also remove public keywords of tests in UtilsTest, as suggested by SonarLint, because they are not needed with Junit 5.
2022-05-28 12:00:46 +02:00
TiA4f8R f6ec7f9a61
Update DefaultStreamExtractorTest and SoundcloudStreamExtractorTest to support changes made in Stream classes 2022-05-28 12:00:45 +02:00
TiA4f8R 7477ed0f3d
[YouTube] Add ability to generate manifests of progressive, OTF and post live streams
A new class has been added to do so: YoutubeDashManifestCreator.
It relies on a new class: ManifestCreatorCache, to cache the content, which relies on a new pair class named Pair.
Results are cached and there is a cache per delivery type, on which cache limit, clear factor, clearing and resetting can be applied to each cache and to all caches.
Look at code changes for more details.
2022-05-28 12:00:45 +02:00
TiA4f8R a857684442
Apply changes in YoutubeStreamExtractor
Extract post live DVR streams as post live streams instead of live streams.

A new class has been in order to improve code: ItagInfo, which stores an itag, the content (URL) extracted and if its an URL or not.
A functional interface has been added in order to abstract the stream building: StreamBuilderHelper.
Also add the cver parameter added by the desktop web client on the corresponding streams (a new method has been added in YoutubeParsingHelper to check this and another for Android streams).

Some code in these classes has been also refactored/improved/optimized.
2022-05-28 12:00:44 +02:00
TiA4f8R 4330b5f7be
Add POST_LIVE_STREAM and POST_LIVE_AUDIO_STREAM stream types
This allows the extractor to determine if a content is an ended audio or video livestream.
2022-05-28 12:00:43 +02:00
TiA4f8R 881969f1da
Apply changes in all StreamExtractors except YouTube's one and fix extraction of PeerTube audio streams as video streams
Some code in these classes has been also refactored/improved/optimized.
Also fix the extraction of PeerTube audio streams as video streams, which are now returned as audio streams.
2022-05-28 12:00:43 +02:00
TiA4f8R d5f3637fc3
[YouTube] Return more values returned inside the ItagItems of the player response and deprecate use of public audio and video fields
These fields can be now replaced by a getter and a setter.

New fields have been added and will allow the creation of DASH manifests for OTF and ended livestreams. There are:
- contentLength;
- approxDurationMs;
- targetDurationSec;
- sampleRate;
- audioChannels.
2022-05-28 12:00:42 +02:00
TiA4f8R 7c67d46e09
Move DashMpdParser to the YouTube package and fix extraction of streams
DashMpdParser is only working with YouTube streams, as it uses the ItagItem class.
Also update creation of AudioStreams and VideoStreams objects.
2022-05-28 12:00:41 +02:00
TiA4f8R ad993b920f
Remove fetching of the DASH manifest extracted when getting information of a content with StreamInfo
DashMpdParser is only working with YouTube streams, as it uses the ItagItem class.

Also improve code and comments of StreamInfo (especially final use where possible).
2022-05-28 12:00:41 +02:00
TiA4f8R 2f061b8dbd
Add support of other delivery methods than progressive HTTP in Stream classes
Stream constructors are now private and streams can be constructed with new Builder classes per stream class. This change has been made to prevent creating and using several constructors in stream classes.

Some default cases have been also added in these Builder classes, so not everything has to be set, depending of the service and the content.
2022-05-28 12:00:27 +02:00
Stypox 1dc80957d8
Merge pull request #848 from TiA4f8R/fix_yt_music_website_extraction_eu
[YouTube] Fix extracting YouTube Music client version and key in the EU
2022-05-22 22:05:05 +02:00
TiA4f8R c34b5e3a8b
[YouTube] Fix extraction of YouTube Music client version and API key when using YouTube Music's website in EU
Google returns now the consent page of YouTube for YouTube Music in EU, which can be also avoided by adding the ucbcb parameter to the URL with the value 1 ("?ucbcb=1").
2022-05-15 11:20:06 +02:00
litetex 2015eb374a Removed more unused methods 2022-05-09 21:05:03 +02:00
litetex f69b0ff77b Remove unused methods 2022-05-09 20:59:25 +02:00
Stypox 23fa31a1ec
Merge pull request #843 from TiA4f8R/fix-checkstyle-issues
Remove Checkstyle suppressions file and fix Checkstyle issues
2022-05-02 22:07:27 +02:00
TiA4f8R 3c3cd78676
Remove Checkstyle suppressions file and fix Checkstyle issues introduced in 24e8399 and 8c1041d
The Checkstyle suppressions file is now replaced by // CHECKSTYLE:OFF and // CHECKSTYLE:ON comments.
2022-05-02 21:51:25 +02:00
Stypox 2e1c5c119d
Merge pull request #822 from Stypox/more-refactors
More refactors
2022-05-02 19:03:54 +02:00
Stypox 598ebb92ea
Merge pull request #839 from TeamNewPipe/bandcamp/extract-length
Bandcamp: extract stream length
2022-05-02 15:49:41 +02:00
litetex 5db4d1faf3
Merge pull request #782 from litetex/cleanup-yt-stream-extractor
Cleanup of ``YoutubeStreamExtractor`` and some related classes
2022-05-01 16:44:11 +02:00
litetex fe30eb43a9 Cleanup ``YoutubeStreamExtractor`` and some related classes
* Fixed obvious sonar(lint) warnings
* Abstracted some code (get*Streams)
* Used some new lines to make code better readable
* Chopped down brace-jungle in some methods
* Use StandardCharset (Java 8 4tw)
2022-05-01 16:39:07 +02:00
Stypox c2b5370517
Apply suggestions: improve switch and use EMPTY_STRING 2022-04-30 16:39:51 +02:00
Stypox 7c78c39230
Merge pull request #821 from litetex/cleanup-TimeAgoParser-java
Cleanup ``TimeAgoParser``
2022-04-30 16:20:09 +02:00
Stypox ac1c22d81c
Release NewPipe Extractor 0.22.1 2022-04-28 11:05:52 +02:00
Stypox 3f5c8962ac
Merge pull request #841 from TiA4f8R/yt-respect-order-of-clients-for-streams
[YouTube] Use correct order of clients to get better streams first
2022-04-28 10:58:49 +02:00
TiA4f8R 9f9af35adb
[YouTube] Fix regression introduced in the order of streams used when adding more parameters to InnerTube requests, using the iOS client for livestreams and more 2022-04-25 20:23:04 +02:00
Fynn Godau c38c016de5 Bandcamp: extract stream length 2022-04-24 21:24:19 +02:00
litetex 5f8f3929bd
Merge pull request #838 from Nickoriginal/update_user_agent
Update User-Agent in NPE
2022-04-24 14:52:18 +02:00
Nickoriginal b5f3f9eb90
Update USER_AGENT to match the latest Firefox 2022-04-24 00:27:23 +03:00
litetex 9f665db4e1
Merge pull request #828 from TeamNewPipe/dependabot/github_actions/actions/setup-java-3
Bump actions/setup-java from 2 to 3
2022-04-23 00:09:44 +02:00
Stypox 52fa2d939a
Fix javadoc formatting error causing deployment to fail 2022-04-16 17:07:07 +02:00
Stypox 95a4b05acc
Release NewPipe Extractor 0.22.0 2022-04-16 16:57:58 +02:00
Tobi b77c72fb88
Merge pull request #832 from Stypox/hotfix-decrypter
Fix YouTube throttling decrypter function parsing
2022-04-15 13:35:10 +02:00
Stypox dcb7483dcf
Fix YouTube throttling decrypter function parsing 2022-04-15 13:10:19 +02:00
Robin bebdc55ad4
Merge pull request #829 from TiA4f8R/yt-subtitles-age-restricted-videos-fix
[YouTube] Extract subtitles for age-restricted videos
2022-04-14 10:31:16 +02:00
TiA4f8R ef49cd0007
[YouTube] Extract subtitles for age-restricted videos
Subtitles of age-restricted videos can be extracted since the InnerTube API migration, so there is no reason to not extract them anymore.
2022-04-11 22:09:56 +02:00
dependabot[bot] 00b51a2bf8
Bump actions/setup-java from 2 to 3
Bumps [actions/setup-java](https://github.com/actions/setup-java) from 2 to 3.
- [Release notes](https://github.com/actions/setup-java/releases)
- [Commits](https://github.com/actions/setup-java/compare/v2...v3)

---
updated-dependencies:
- dependency-name: actions/setup-java
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-04-11 09:16:36 +00:00
TiA4f8R 2e92d718a2
Merge pull request #780 from TiA4f8R/yt-more-params-innertube-requests
[YouTube] Add more parameters to InnerTube requests, use the iOS client for livestreams and fix extraction of embeddable age-restricted videos and contents with a warning before playback
2022-04-09 18:15:18 +02:00
TiA4f8R b30e341559
Add link to learn more about the issue which makes YoutubeStreamExtractorRelatedMixTest.testRelatedItems test disabled 2022-04-04 20:36:59 +02:00
TiA4f8R 70812fa611
Update YouTube stream mocks and disable YoutubeStreamExtractorRelatedMixTest
Mixes seems to be not given by YouTube anymore if you use a PENDING consent cookie value.
As mocks needs to updated, the test is always failing because of this change.
2022-04-02 19:28:49 +02:00