Commit Graph

1633 Commits

Author SHA1 Message Date
TiA4f8R 448b68700c
Use final in exceptions and add AgeRestrictedException
AgeRestrictedException will be thrown only if the reason message equals to "Sign in to confirm your age" and if the age limit is 18.
2021-03-05 16:38:49 +01:00
TobiGr 22e6f33f0a
Ignore SoundCloudGeoRestrictedTest until HLS stream extraction is added
The ContentNotSupportedException is thrown because no supported audio streams where extracted. However, SoundCLoud does not check, whether there are any streams available. 
This commit should be reverted in #526
2021-03-05 16:38:47 +01:00
TobiGr 21158744b1
Add expectedDescriptionIsEmpty() to DefaultStreamExtractorTest
Also check if related streams are empty if they are expected to be empty.
2021-03-05 16:38:46 +01:00
TiA4f8R f15d7837a1
Fix SoundCloud tests 2021-03-05 16:38:45 +01:00
TiA4f8R 4e45aef2b3
Fix SoundCloud CreativeCommonsPlaysWellWithOthers test 2021-03-05 16:38:43 +01:00
TiA4f8R 59d6d3f04e
Fix YouTube tests 2021-03-05 16:38:42 +01:00
TiA4f8R 3925204658
Add tests and rename SoundcloudGoPlusException to SoundcloudGoPlusContentException 2021-03-05 16:38:40 +01:00
TiA4f8R 771bb1a2cb
Add three new exceptions for unavailable YouTube contents
These exceptions are thrown on a test with the error messages text, because YouTube returns only "UNPLAYABLE" status in most error cases.
Tests are based with English strings, so changing the lang used by
extractor will throw the generic exception (ContentNotAvailableException).
2021-03-05 16:38:39 +01:00
TiA4f8R 35325d980d
Add GeographicRestrictionException and SoundCloudGoPlusException in NewPipe Extractor to be able to display different error messages
This commit adds two new exceptions in NewPipe Extractor: GeographicRestrictionException and SoundCloudGoPlusException (which extend to ContentNotAvailableException). These exceptions allow showing different error messages to user when a content isn't available in his/her/its country (only used for now by SoundCloudStreamExtractor) or when the content is a SoundCloud Go+ track.
2021-03-05 16:38:37 +01:00
TobiGr cb07ffa1eb Release 0.20.11 2021-03-05 15:35:11 +01:00
Tobi bc0cda68d1
Merge pull request #570 from TiA4f8R/fix-yt-channels-playlists
Fix YouTube channels contination
2021-03-05 15:26:49 +01:00
Tobi a3c6fceef5
Merge pull request #573 from B0pol/comments-performance
[youtube] improve comments extraction performance
2021-03-05 14:46:27 +01:00
bopol ff5273b882
Update extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/extractors/YoutubeCommentsExtractor.java
Co-authored-by: Tobi <TobiGr@users.noreply.github.com>
2021-03-05 14:39:01 +01:00
Tobi ec1127dd6a
Merge pull request #571 from TeamNewPipe/soundcloud_client_id
[SoundCloud] Update client_id and fix extraction
2021-03-05 14:21:14 +01:00
bopol ed850d0688 [youtube] improve comments extraction performance
- do not parse responseBody twice for continuation
instead try to get commentsTokenInside with the new pattern ("sectionListRenderer")
and try again with the old pattern ("commentSectionRenderer") on failure
- do not unescape responseBody multiple times
   -> parse responseBody less times
2021-03-05 13:37:08 +01:00
TobiGr 440a808b8a Update mocks for channels 2021-03-05 13:13:36 +01:00
Tobi b608587e4d
Merge pull request #572 from golfinq/fixed-comments
Fixes missing comments under videos
2021-03-05 11:42:40 +01:00
golfinq 05213175cd
minor edit, made variable final 2021-03-05 02:49:15 -05:00
golfinq 8bceb57a6c
formatting fix 2021-03-05 02:46:06 -05:00
golfinq ae48bdea4c
Updated to reflect variable name change 2021-03-05 02:40:36 -05:00
TiA4f8R b27efdc9a4
Remove clickTrackingParams in getNextPage method of YoutubeChannelExtractor
Because it's unused with the new request.

Co-authored-by: bopol <bopol@e.email>
2021-03-04 22:50:20 +01:00
TobiGr 2b671b15ce [YouTube] Use new continuation API for channels
Co-authored-by: TiA4f8R <74829229+tia4f8r@users.noreply.github.com>
2021-03-04 21:47:16 +01:00
TobiGr 33594d4aff [SoundCloud] Fix client id extraction
SoundCloud changed their Javascript structure. That caused the client_id to be in the middle of the file (at byte ~43000 ). To have a little buffer, we now fetch the first 50000 bytes.
2021-03-04 21:32:34 +01:00
TobiGr fb2f37d223 [SoundCloud] Update hardcoded client id 2021-03-04 21:25:41 +01:00
XiangRongLin 9256b3b848
Merge pull request #567 from XiangRongLin/playlist_continuations
Playlist continuations
2021-03-04 19:05:32 +01:00
XiangRongLin 506cc5fbe9 regenerate mocks 2021-03-04 19:01:58 +01:00
XiangRongLin 03b00ff1d6 Move toJsonArray and toJsonObject to JsonUtils 2021-03-04 18:58:51 +01:00
FireMasterK 668b080b3a Use dynamic key. 2021-03-04 08:45:14 +01:00
XiangRongLin 43b46bd408 Use new youtube continuations api for playlists
Requires sending a POST request instead of GET.
clientName and clientVersion, which were required as headers previously now need to be part of the request payload.
continuation id also needs to be part of request body.

quick and dirty solution.
2021-03-03 19:49:26 +01:00
XiangRongLin beb05bd05c Replace test url, because playlist was deleted 2021-03-03 19:46:14 +01:00
XiangRongLin e13e237392 Fix typo and reword some explanations 2021-02-26 17:48:03 +01:00
Tobi 5d594cfded
Merge pull request #549 from FireMasterK/gradle-update
Update gradle wrapper.
2021-02-24 15:59:28 +01:00
Tobi 965a66bc87
Merge pull request #557 from TiA4f8R/fix-encoding-errors-windows
Set extractor encoding to UTF-8
2021-02-24 15:49:58 +01:00
TiA4f8R b2cf41496d
Set extractor encoding to UTF-8
When compiling the extractor on Windows, you get more than 100 errors with the message error: unmappable character for encoding Cp1252, especially in timeago-parser tasks. This commit fixes it by setting the encoding for all compileJava and compileTestJava tasks to UTF-8.
2021-02-23 14:38:20 +01:00
TobiGr a1688fe953 Move BandcampExtractorHelper.getJsonData(String, String) to JsonUtils 2021-02-21 13:51:12 +01:00
TobiGr 70814dcfef Fix Utils.nonEmptyAndNullJoin
When using the index here, it the index needs to be decremented once an element is removed. To cirecumvent this, the native Collections.removeIf() method is used.
2021-02-21 12:51:47 +01:00
TobiGr b9e8ee8450 Rename BandcampExtractorHelper.smartConcatenate(String[], String) to Utils.nonEmptyAndNullJoin(String, String[]) 2021-02-21 12:35:14 +01:00
TobiGr c07db80ef0 Add BASE_URL and BASE_API_URL to BandcampExtractorHelper 2021-02-21 12:15:45 +01:00
TobiGr adde4332d1 Code improvements 2021-02-20 23:58:54 +01:00
FireMasterK 42bdd8adc5
Update to fix failing mock tests. 2021-02-20 17:14:41 +05:30
FireMasterK e8bccfaf5d
Change UA to privacy.resistFingerprinting. 2021-02-20 17:14:41 +05:30
bopol a9aa385bb3
Merge pull request #554 from B0pol/test-verified
use DownloaderFactory instead of DownloaderTestImpl for search Channe…
2021-02-20 12:12:43 +01:00
bopol 0c831afa7b use DownloaderFactory instead of DownloaderTestImpl for search ChannelVerified 2021-02-20 11:41:30 +01:00
TobiGr ea120a4637 remove print stacktrace
use interface
2021-02-20 00:23:45 +01:00
XiangRongLin adf9d7d10f Add reason field to MockOnly
This enforces developers to document why a test is skipped
2021-02-19 18:54:18 +01:00
XiangRongLin ea52030613 Add MockOnlyRule to allow skipping specific tests based on downloader 2021-02-19 18:54:17 +01:00
TobiGr 02920fafa8 Add isUploaderVerified() 2021-02-19 15:51:23 +01:00
TobiGr 91e9309486 Merge remote-tracking branch 'origin/dev' into bandcamp 2021-02-19 13:37:45 +01:00
TobiGr fa61b864f2 Code improvements 2021-02-19 13:37:24 +01:00
TobiGr 98268e351c Move radio URL check into a function 2021-02-19 13:35:44 +01:00