Commit Graph

1966 Commits

Author SHA1 Message Date
TiA4f8R 3d38459cf3
[YouTube] Reduce InnerTube response sizes by adding the prettyPrint parameter with the false value
InnerTube responses return pretty printed responses, which increase responses' size for nothing.

By using the prettyPrint parameter on requests and setting its value to false, responses are not pretty printed anymore, which reduces responses size, and so data transfer and processing times.
This usage has been recently deployed by YouTube on their websites.
2022-03-27 20:52:40 +02:00
litetex 349ba8db7f
Improve tests and randomness
- Use the existing RNG inside YoutubeParsingHelper
- Deduplicated test-setup for YouTube tests
- Minor improvements
2022-03-27 20:52:38 +02:00
TiA4f8R 52376949e5
Add setSeedForVideoTests method in YoutubeStreamExtractor tests
In order to use still use mocks with the generation of random strings in player requests, we need to use YoutubeParsingHelper.setSeedForVideoTests() method in every stream test.
2022-03-27 20:51:39 +02:00
TiA4f8R d0d91e6690
Adress requested changes 2022-03-27 20:51:39 +02:00
TiA4f8R b6bc521f0d
[YouTube] Update client versions again 2022-03-27 20:51:38 +02:00
TiA4f8R 26f93f5bb0
[YouTube] Extract streams of livestreams from the iOS client and disabled the Android client for livestreams
The iOS client is only enabled for livestreams and the Android client is now only enabled for videos, both by default.

A way to force, or not, the fetch of both clients have been added with two new static methods in YoutubeStreamExtractor.
2022-03-27 20:51:38 +02:00
TiA4f8R 7d07924de8
[YouTube] Try to use lighter requests when extracting client version and key from YouTube and YouTube Music
This is done by fetching https://www.youtube.com/sw.js for YouTube and https://music.youtube.com/sw.js for YouTube Music.

Two new methods in Utils class have been added which allow to try to get a match of regular expressions in a string array, or a Pattern array, on a content, on a specific index or 0.
Also some code refactoring has been made in this class.
2022-03-27 20:51:38 +02:00
TiA4f8R 05b7fee23b
[YouTube] Add the cpn param to playback requests and try to spoof better the Android client
The cpn param, aka the content playback nonce param, is a parameter sent by YouTube web client in videoplayback requests, and for some of them, in the player request body. This PR adds it everywhere.

For the desktop/WEB client, some params were missing from the playbackContext object, which seemed (or not) to make YouTube throttle streams extracted from the WEB client. This PR adds them.

Fingerprinting on the WEB client basing on the client version used is not possible anymore, because the latest client version is extracted at the first time of a YouTube request on a session which require the extractor to fetch again the website (and this may come back the reCaptcha issues again unfortunately, but it seems there is no other way to get it).

For the Android client, the video id is now also sent as a query parameter, like a 12 characters string, in the t query parameter, in order to spoof better this client. Researches need to be done on this parameter, unique to each request, and how it is generated by clients.

This commit also fixes a small bug with the Android User-Agent string.

Some code improvements have been also made.
2022-03-27 20:51:38 +02:00
TiA4f8R 83f374bff1
[YouTube] Update client versions and fix a bug when using resetClientVersionAndKey method
The boolean keyAndVersionExtracted in YoutubeParsingHelper was not set to false when resetting the client version and the key, which makes the extractor uses null on the next getting of the client version or the key if the clientVersion and the key were extracted before.
Also update client versions.
2022-03-27 20:51:38 +02:00
litetex 5a18730845
Merge pull request #813 from litetex/fix-test-2022-03
Fixed tests (+ Youtube shorts in channels)
2022-03-26 21:09:10 +01:00
litetex af82edf9dc Fix checkstyle problems 2022-03-26 20:54:20 +01:00
litetex 29408f9356 ``YoutubeMusicSearchExtractorTest$Suggestion`` has no mock data 2022-03-26 20:52:28 +01:00
litetex fcee247f9f Switched to mockonly tests
* Made the reason why the tests are mockonly mandatory
2022-03-26 20:52:28 +01:00
litetex 0fceb4686b Added missing mock data 2022-03-26 20:52:27 +01:00
litetex 70b20f0d6b Updated mock data after conflicts 2022-03-26 20:52:27 +01:00
litetex 6a6c9359af Fixed kurzgesagt test - They changed their description... 2022-03-26 20:52:27 +01:00
litetex 26596215fa Refactored ``MediaCCCRecentListExtractorTest`` 2022-03-26 20:52:26 +01:00
litetex 53962bfd7d Disabled suggestion test for now
Has to be done once YT's backed works again...
2022-03-26 20:52:26 +01:00
litetex 66dc5e8bb8 API hardening against changes 2022-03-26 20:52:26 +01:00
litetex 5d58156cde Update mock data 2022-03-26 20:52:25 +01:00
litetex 7598b40957 Workaround for incorrect duration for "YT shorts" videos in channels
As a workaround 0 is returned as duration for such videos.
See also https://github.com/TeamNewPipe/NewPipe/issues/8034
2022-03-26 20:52:24 +01:00
litetex 164e21b5af Fixed ``MediaCCCRecentKiosk``
Ignore faulty data/items (with duration <= 0)
2022-03-26 20:52:23 +01:00
litetex 49dfdae5d2 Removed useless throw declaration 2022-03-26 20:51:38 +01:00
litetex 69a58fd3d1 Don't use sysout 2022-03-26 20:51:37 +01:00
litetex 639be7adda Minimized some code 2022-03-26 20:51:37 +01:00
litetex 2bd4299563 Fixed test: Peertube - Use new comments and videos
Old comments/videos got deleted
2022-03-26 20:49:39 +01:00
litetex 606a386a98 Fix tests: What's peertube has less comments now 2022-03-26 20:49:39 +01:00
litetex 4d1a1c8fb8 Better test for ``MediaCCCRecentListExtractorTest``
* Use assertAll
* Show which item is affected
2022-03-26 20:49:38 +01:00
litetex ba43dbaa28 Fix Tests: The channel lost abos 2022-03-26 20:49:38 +01:00
litetex ff436e5740 Fixed test: YT mix playlist 2022-03-26 20:49:38 +01:00
litetex bb49f7d857
Merge pull request #817 from Stypox/checkstyle
Add checkstyle
2022-03-26 20:46:49 +01:00
litetex 9c07e8a664 Fix useage of wrong object 2022-03-26 20:17:50 +01:00
litetex 93d6e5c4d5 CI: Upload test reports when failure occurs 2022-03-26 19:57:17 +01:00
litetex 804e57004f Fixed new checkstyle problems from dev 2022-03-26 19:46:10 +01:00
litetex 33347ac18b Removed unused methods
``contentFilters`` and ``sortfilter`` are get inside the ``ListLinkHandler`` and not the ``ListLinkHandlerFactory``
 ``ListLinkHandlerFactory`` only passes these values through when ``fromQuery`` is called
2022-03-26 19:43:11 +01:00
litetex ec5b54c38b Removed unused class 2022-03-26 19:43:10 +01:00
litetex 8771af7ba5 Restored original naming 2022-03-26 19:43:09 +01:00
litetex 01cfde0285 Fixed typo 2022-03-26 19:43:09 +01:00
Stypox bdadcfa1f7 Legitimately suppress remaining checkstyle warnings 2022-03-26 19:43:08 +01:00
Stypox 740a37a2de [YouTube] Fix checkstyle issues 2022-03-26 19:42:40 +01:00
Stypox 9dc17cd1ca [Soundcloud] Fix checkstyle issues 2022-03-26 19:40:20 +01:00
Stypox 9ab32cb2e7 [Peertube] Fix checkstyle issues 2022-03-26 19:40:19 +01:00
Stypox 9f7e06c817 [MediaCCC] Fix checkstyle issues 2022-03-26 19:40:18 +01:00
Stypox 3a94839359 [Bandcamp] Fix checkstyle issues 2022-03-26 19:40:17 +01:00
Stypox 08dff33002 Use Java 8 streams in NewPipe class 2022-03-26 19:40:15 +01:00
Stypox c2446ecff0 Use Java 8 streams and deduplicate code in MediaFormat class 2022-03-26 19:40:15 +01:00
Stypox d79e20340c Fix checkstyle issues in root package extractor/
Note: not all issues were fixed because MediaFormat and ServiceList use a specific formatting that makes sense for them
2022-03-26 19:40:14 +01:00
Stypox ca7c63f273 Fix remaining checkstyle issues in utils/ subpackage 2022-03-26 19:40:13 +01:00
Stypox 1d5f22e41f Fix checkstyle issues & more in JsonUtils
Also use Java 8 streams and extract duplicate code to getInstanceOf function
2022-03-26 19:40:13 +01:00
Stypox 87d2834986 Fix checkstyle issues & more in DonationLinkHelper
Also add comment about the class being unused and replace the fixLink function with Utils.stringToUrl()
2022-03-26 19:40:12 +01:00