0438828e36
Add a test for the number of audioStreams in CreativeCommonsPlaysWellWithOthers test
...
It should be only two audio streams for track "Plays Well with Others, Ep 2: What Do an Army of Ants and an Online Encyclopedia Have in Common?" by Creative Commons (https://soundcloud.com/wearecc/plays-well-with-others-ep-2-what-do-an-army-of-ants-and-an-online-encyclopedia-have-in-common ):
- one which is a progressive stream, in MP3 format with a bitrate of 128 kbps
- one which is an HLS stream, in OPUS format with a bitrate of 64 kbps.
2021-03-14 17:54:18 +01:00
3bd08a2880
Adress requested changes and use final where possible in SoundcloudStreamExtractor
...
This commit moved the HLS parsing task to a separate method, did little performance improvements and used final where possible in the SoundcloudStreamExtractor file.
2021-03-14 17:54:17 +01:00
cbacd3c0a5
Add a check to don't show MP3 128kbps stream twice and catch IOException when fetching the HLS Manifest
...
If a progressive stream is present in the transcodings, it's unnecessary to show twice an MP3 128kbps stream so if this is the case, the MP3 HLS stream will be not added to the audioStreams, else it will.
This commit also catch fetching errors in HLS manifests parsing and don't add the corresponding stream if an error occurs.
2021-03-14 17:54:16 +01:00
26f1b4e7dc
Support SoundCloud HLS by using a workaround
...
This commit tries to support SoundCloud HLS streams by parsing M3U manifests, get the last segment URL (in order to get track length) and request a segment URL equals to track's duration so it's a single URL.
2021-03-14 17:54:14 +01:00
2e57a8f24f
[Bandcamp] Fix link handler acceptance behaviour
...
* Test for bandcamp footer instead of meta tag (which is not present on
all pages)
* Accept links to /music, not just /releases
* Correctly handle uppercase URLs
2021-03-14 09:51:30 +01:00
70d9e389b9
[Bandcamp] Add tests for external bandcamp artists
...
Necessary, because the external pages tested before were converted to bandcamp.com pages. See f9d06252f2
2021-03-13 20:51:59 +01:00
f9d06252f2
[Bandcamp] Fix tests
2021-03-13 17:42:39 +01:00
7e6f464407
Merge pull request #509 from TiA4f8R/soundcloud-improvements
...
Add new exceptions to be able to display different error messages in apps
2021-03-07 13:51:15 +01:00
ac51134aed
Merge pull request #546 from XiangRongLin/unignore2
...
Unignore more tests
2021-03-07 13:15:24 +01:00
22fa131922
Merge branch 'dev' into bandcamp
2021-03-05 19:45:37 +01:00
e55284bb8f
Code style improvements
...
Use final where possible in YoutubeStreamExtractor and do some other code style improvements
2021-03-05 16:38:53 +01:00
d0a1041afd
Prevent NullPointerException and use equalsIgnorecase
2021-03-05 16:38:52 +01:00
890cbba625
Update PaidContent exception
2021-03-05 16:38:51 +01:00
da3cfa967d
Handle age-restricted videos
2021-03-05 16:38:50 +01:00
448b68700c
Use final in exceptions and add AgeRestrictedException
...
AgeRestrictedException will be thrown only if the reason message equals to "Sign in to confirm your age" and if the age limit is 18.
2021-03-05 16:38:49 +01:00
22e6f33f0a
Ignore SoundCloudGeoRestrictedTest until HLS stream extraction is added
...
The ContentNotSupportedException is thrown because no supported audio streams where extracted. However, SoundCLoud does not check, whether there are any streams available.
This commit should be reverted in #526
2021-03-05 16:38:47 +01:00
21158744b1
Add expectedDescriptionIsEmpty() to DefaultStreamExtractorTest
...
Also check if related streams are empty if they are expected to be empty.
2021-03-05 16:38:46 +01:00
f15d7837a1
Fix SoundCloud tests
2021-03-05 16:38:45 +01:00
4e45aef2b3
Fix SoundCloud CreativeCommonsPlaysWellWithOthers test
2021-03-05 16:38:43 +01:00
59d6d3f04e
Fix YouTube tests
2021-03-05 16:38:42 +01:00
3925204658
Add tests and rename SoundcloudGoPlusException to SoundcloudGoPlusContentException
2021-03-05 16:38:40 +01:00
771bb1a2cb
Add three new exceptions for unavailable YouTube contents
...
These exceptions are thrown on a test with the error messages text, because YouTube returns only "UNPLAYABLE" status in most error cases.
Tests are based with English strings, so changing the lang used by
extractor will throw the generic exception (ContentNotAvailableException).
2021-03-05 16:38:39 +01:00
35325d980d
Add GeographicRestrictionException and SoundCloudGoPlusException in NewPipe Extractor to be able to display different error messages
...
This commit adds two new exceptions in NewPipe Extractor: GeographicRestrictionException and SoundCloudGoPlusException (which extend to ContentNotAvailableException). These exceptions allow showing different error messages to user when a content isn't available in his/her/its country (only used for now by SoundCloudStreamExtractor) or when the content is a SoundCloud Go+ track.
2021-03-05 16:38:37 +01:00
bc0cda68d1
Merge pull request #570 from TiA4f8R/fix-yt-channels-playlists
...
Fix YouTube channels contination
2021-03-05 15:26:49 +01:00
a3c6fceef5
Merge pull request #573 from B0pol/comments-performance
...
[youtube] improve comments extraction performance
2021-03-05 14:46:27 +01:00
ff5273b882
Update extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/extractors/YoutubeCommentsExtractor.java
...
Co-authored-by: Tobi <TobiGr@users.noreply.github.com>
2021-03-05 14:39:01 +01:00
ec1127dd6a
Merge pull request #571 from TeamNewPipe/soundcloud_client_id
...
[SoundCloud] Update client_id and fix extraction
2021-03-05 14:21:14 +01:00
ed850d0688
[youtube] improve comments extraction performance
...
- do not parse responseBody twice for continuation
instead try to get commentsTokenInside with the new pattern ("sectionListRenderer")
and try again with the old pattern ("commentSectionRenderer") on failure
- do not unescape responseBody multiple times
-> parse responseBody less times
2021-03-05 13:37:08 +01:00
440a808b8a
Update mocks for channels
2021-03-05 13:13:36 +01:00
05213175cd
minor edit, made variable final
2021-03-05 02:49:15 -05:00
8bceb57a6c
formatting fix
2021-03-05 02:46:06 -05:00
ae48bdea4c
Updated to reflect variable name change
2021-03-05 02:40:36 -05:00
b27efdc9a4
Remove clickTrackingParams in getNextPage method of YoutubeChannelExtractor
...
Because it's unused with the new request.
Co-authored-by: bopol <bopol@e.email>
2021-03-04 22:50:20 +01:00
2b671b15ce
[YouTube] Use new continuation API for channels
...
Co-authored-by: TiA4f8R <74829229+tia4f8r@users.noreply.github.com>
2021-03-04 21:47:16 +01:00
33594d4aff
[SoundCloud] Fix client id extraction
...
SoundCloud changed their Javascript structure. That caused the client_id to be in the middle of the file (at byte ~43000 ). To have a little buffer, we now fetch the first 50000 bytes.
2021-03-04 21:32:34 +01:00
fb2f37d223
[SoundCloud] Update hardcoded client id
2021-03-04 21:25:41 +01:00
506cc5fbe9
regenerate mocks
2021-03-04 19:01:58 +01:00
03b00ff1d6
Move toJsonArray and toJsonObject to JsonUtils
2021-03-04 18:58:51 +01:00
668b080b3a
Use dynamic key.
2021-03-04 08:45:14 +01:00
43b46bd408
Use new youtube continuations api for playlists
...
Requires sending a POST request instead of GET.
clientName and clientVersion, which were required as headers previously now need to be part of the request payload.
continuation id also needs to be part of request body.
quick and dirty solution.
2021-03-03 19:49:26 +01:00
beb05bd05c
Replace test url, because playlist was deleted
2021-03-03 19:46:14 +01:00
e13e237392
Fix typo and reword some explanations
2021-02-26 17:48:03 +01:00
a1688fe953
Move BandcampExtractorHelper.getJsonData(String, String) to JsonUtils
2021-02-21 13:51:12 +01:00
70814dcfef
Fix Utils.nonEmptyAndNullJoin
...
When using the index here, it the index needs to be decremented once an element is removed. To cirecumvent this, the native Collections.removeIf() method is used.
2021-02-21 12:51:47 +01:00
b9e8ee8450
Rename BandcampExtractorHelper.smartConcatenate(String[], String) to Utils.nonEmptyAndNullJoin(String, String[])
2021-02-21 12:35:14 +01:00
c07db80ef0
Add BASE_URL and BASE_API_URL to BandcampExtractorHelper
2021-02-21 12:15:45 +01:00
adde4332d1
Code improvements
2021-02-20 23:58:54 +01:00
42bdd8adc5
Update to fix failing mock tests.
2021-02-20 17:14:41 +05:30
e8bccfaf5d
Change UA to privacy.resistFingerprinting.
2021-02-20 17:14:41 +05:30
0c831afa7b
use DownloaderFactory instead of DownloaderTestImpl for search ChannelVerified
2021-02-20 11:41:30 +01:00
ea120a4637
remove print stacktrace
...
use interface
2021-02-20 00:23:45 +01:00
adf9d7d10f
Add reason field to MockOnly
...
This enforces developers to document why a test is skipped
2021-02-19 18:54:18 +01:00
ea52030613
Add MockOnlyRule to allow skipping specific tests based on downloader
2021-02-19 18:54:17 +01:00
02920fafa8
Add isUploaderVerified()
2021-02-19 15:51:23 +01:00
91e9309486
Merge remote-tracking branch 'origin/dev' into bandcamp
2021-02-19 13:37:45 +01:00
fa61b864f2
Code improvements
2021-02-19 13:37:24 +01:00
98268e351c
Move radio URL check into a function
2021-02-19 13:35:44 +01:00
5333d8a98b
Add more data.
2021-02-19 00:55:03 +05:30
1a322ad8ed
Add uploader verified by service extraction
2021-02-18 16:36:40 +01:00
c24afa2cbb
Permalink URL.
2021-02-18 20:16:53 +05:30
8d54401233
Add fps field as well.
2021-02-18 20:16:53 +05:30
5b70645d85
Add data to muxed streams aswell.
2021-02-18 20:16:53 +05:30
e975d33fbe
Change jdoc / comments.
2021-02-18 20:16:53 +05:30
f31b2a68fd
use getters/setters.
2021-02-18 20:16:53 +05:30
d4945ac55c
Constructor cleanup.
2021-02-18 20:16:52 +05:30
3f6a601be8
Fix jdoc / comments.
2021-02-18 20:16:52 +05:30
a7c9905183
Fix requested changes.
2021-02-18 20:16:52 +05:30
0c0f2d74bc
Make fields private
...
Co-authored-by: bopol <bopol@e.email>
2021-02-18 20:16:52 +05:30
d1054338d3
Fix error with livestreams.
2021-02-18 20:16:52 +05:30
11eb4932f4
Add data to respective classes.
2021-02-18 20:16:52 +05:30
525e345ed8
remove duplicate field.
2021-02-18 20:16:52 +05:30
e844d2aed3
Add dash fields to the ItagItem.
2021-02-18 20:16:52 +05:30
64f6b0478f
Merge pull request #545 from XiangRongLin/comment_reset
...
Add comment explaining YoutubeParsingHelper.resetClientVersionAndKey
2021-02-18 15:14:03 +01:00
a64579da2f
Unignore some tests that work on Linux
2021-02-17 20:31:26 +01:00
b99cb8624f
Unignore PublicBroadcasterTest
...
Tests only work on linux, mocks have to be generated on linux too
2021-02-17 20:31:26 +01:00
37366c65ca
Remove encoding of '&' to '&' in test and unignore it
2021-02-17 20:31:26 +01:00
c327922cc0
Add explanations to broken tests
2021-02-17 20:31:26 +01:00
26d50aca05
Add comment explaining YoutubeParsingHelper.resetClientVersionAndKey
2021-02-17 19:21:39 +01:00
54aa5b3042
Use propper structure in KioskExtractors
...
Made BandCampRadioExtractor a Kiosk which holds StreamInfoItems and not InfoItems.
2021-02-16 21:09:42 +01:00
54b8e54f80
Fix potential NPE
2021-02-16 20:59:13 +01:00
16973126a0
Use Collections.emptyList() instead of Collections.EMPTY_LIST or new ArrayList()
2021-02-16 20:25:48 +01:00
84380e4c3a
Merge pull request #536 from B0pol/watchendpoints
...
support /watch/, /v/ and /w/ subpaths
2021-02-16 20:23:27 +01:00
41f689b099
Update extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/linkHandler/YoutubeStreamLinkHandlerFactory.java
...
Co-authored-by: Stypox <stypox@pm.me>
2021-02-16 19:17:10 +01:00
a9594f531f
Merge pull request #543 from XiangRongLin/unignore1
...
Unignore and fix some tests
2021-02-16 18:00:16 +01:00
a28e0226c5
Merge pull request #519 from XiangRongLin/better_logging_error_handling
...
Better logging, error handling, doc for mock tests
2021-02-16 15:06:06 +01:00
05cb22c654
Unignore Gronkh.testDescription
...
Remove special character ü from assertion
2021-02-16 08:59:35 +01:00
d9b4c2779f
Unignore YoutubeSearchQHTest.testRegularValues
2021-02-16 08:59:35 +01:00
137272b7a4
Unignore YoutubeCommentsExtractorTest.EmptyComment
...
Mock files were missing
2021-02-16 08:59:34 +01:00
5bf9fddba9
Code improvements
2021-02-15 22:45:17 +01:00
e062c8cb0d
Merge branch 'dev' into bandcamp
2021-02-15 22:09:41 +01:00
7a3d9bdb7d
Description: add .equals() and .hashCode()
2021-02-15 18:51:47 +01:00
557934cb17
use EMPTY_STRING
2021-02-15 18:51:46 +01:00
9ca52ca68a
use only one constant UTF-8
2021-02-15 18:47:54 +01:00
50e5718bd1
Merge pull request #518 from XiangRongLin/generate_mocks
...
Use and generate mocks for youtube tests
2021-02-15 06:39:41 +01:00
d7486f251a
Merge pull request #538 from B0pol/soundcloud-georestriction
...
Soundcloud: fix charts extraction when georestricted
2021-02-14 23:52:27 +01:00
432c68cdec
Update okhttp from 3.12.11 to 3.12.13
...
Changelog https://square.github.io/okhttp/changelog_3x/#version-31213
2021-02-14 23:44:19 +01:00
a88b2e6af4
Undo formatting of YoutubeStreamExtractorDefaultTest
2021-02-14 14:57:23 +01:00
8d89c82caa
Soundcloud: fix charts extraction when georestricted
2021-02-13 22:28:31 +01:00
ee2012c948
rename folder to subpath
2021-02-13 22:25:46 +01:00
8db40db4de
use substring instead of split
2021-02-13 22:20:05 +01:00
81317cf6bc
youtube: support /w/ folder
2021-02-13 22:20:05 +01:00
f15c0fcfed
support /watch/ and /v/ folders
2021-02-13 22:20:05 +01:00
971128c464
Ignore failing tests
2021-02-13 20:28:03 +01:00
92dea0806a
Regenerate json files for youtube tests
2021-02-13 20:28:03 +01:00
d6b9930c18
Use DownloaderFactory in YoutubeCommentsExtractorTest.Pinned
2021-02-13 20:12:39 +01:00
b555f38fa5
Use real downloader for YoutubeSubscriptionExtractorTest
...
It doesn't make network requests anyway
2021-02-13 20:12:39 +01:00
f352b0c7af
YoutubeMusicSearchExtractorTest can't be used with mocks
...
Requires further research
2021-02-13 20:00:51 +01:00
f45b5610bf
Unignore YoutubeChannelLocalizationTest
...
With mocks time is not a concern anymore
2021-02-13 20:00:51 +01:00
19737e06ee
Use DownloaderFactory in YoutubeChannelExtractorTest
...
KurzgesagtAdditional needed to be extracted because it ran a duplicated request.
2021-02-13 20:00:51 +01:00
84eb4b30c9
Use DownloaderFactory in YouTube tests
2021-02-13 20:00:51 +01:00
b43f3474f5
Make resetClientVersionAndKey public
2021-02-13 20:00:49 +01:00
2859c7ba5c
add comment explaining why .substring(23)
2021-02-13 12:10:41 +01:00
73353996d7
fix more tests
2021-02-12 23:18:38 +01:00
a931e31239
fix tests
2021-02-12 22:36:18 +01:00
d116680ea3
Merge branch 'master' into dev
2021-02-12 22:26:35 +01:00
b0f356dd76
YT channels: fix continuation
2021-02-12 14:39:09 +01:00
fcdb50b825
make getInfo from Extractor public
...
In StreamInfo and CommentsInfo
2021-02-09 14:58:43 +01:00
44c54d403a
fix Invidious channels being accepted by YoutubeStreamLinkHandlerFactory ( #527 )
...
fixes https://github.com/TeamNewPipe/NewPipeExtractor/issues/524 , see it for more details
2021-02-05 18:57:58 +01:00
1414a6f178
Fix extraction of YouTube chapters ( #531 )
...
YouTube changed the name of the chapters in the JSON from engagement-panel-macro-markers to engagement-panel-macro-markers-description-chapters, so extracting chapters doesn't work.
2021-02-03 22:07:34 +01:00
bfa639950e
Merge pull request #528 from Stypox/ytmusic-0
...
[YouTube Music] Fix extracting search item view/subscriber count when = 0
2021-01-29 23:23:23 +01:00
d728c4fad0
Remove unused imports
...
I searched for "StandardCharsets" to prevent possible bugs with Android KitKat. The import is unused and is therefor removed.
There are other usages of the StandardCharsets class in the test implementation, but those are not relevant.
2021-01-29 23:16:40 +01:00
a64dfd7343
[YouTube Music] Fix extracting search item view/subscriber count when = 0
2021-01-29 16:44:24 +01:00
2662ceca22
rename getHeartedByCreator and getPinned to isPinned and isHeartedByCreator
2021-01-23 13:06:07 +01:00
66e4eb2f96
extract whether the comment is pinned
2021-01-22 23:59:22 +01:00
67ddfefdc6
keep invidious instances list in the same order
...
as suggested by @TobiGr
2021-01-22 20:58:14 +01:00
53d3f7989b
update invidious instances list
2021-01-22 19:30:25 +01:00
88e4c8667b
Add comments to RecordingDownloader about usage
2021-01-17 19:53:20 +01:00
eecfe09f2c
Check for non-existent folder in MockDownloader
2021-01-17 19:52:58 +01:00
07570554b6
Fix build and optimize imports
2021-01-15 21:55:40 +01:00
78c2113094
Merge remote-tracking branch 'origin/dev' into bandcamp
2021-01-15 21:49:58 +01:00
b9ba95614b
Merge pull request #514 from XiangRongLin/test_jvm_property
...
Pass on gradle JVM system property to test JVM
2021-01-15 20:44:23 +01:00
a6cfe11ae9
Pass on gradle JVM system property to test JVM
2021-01-15 20:37:14 +01:00
6888e7773f
Ignore all failing tests
2021-01-15 20:11:40 +01:00
7c774c84ce
Merge pull request #505 from TeamNewPipe/meida.ccc.de_search_fix
...
[media.ccc.de] Fix NPE in search results if they contain a future talk
2021-01-15 14:16:59 +01:00
c2c4d97efd
Merge pull request #482 from XiangRongLin/mock_mix_pl_test
...
Mock mix pl tests
2021-01-15 12:54:48 +01:00
4f81d9d689
Regenerate mock files
2021-01-15 09:51:23 +01:00
951159f75f
Merge pull request #504 from TeamNewPipe/media.ccc.de_recent_order
...
[media.ccc.de] Recent kiosk: order streams by upload date
2021-01-14 23:31:09 +01:00
cc51c5f6a3
Merge pull request #507 from vkay94/add-interval-to-stream-frames
...
Add durationPerFrame to Frameset
2021-01-14 20:27:40 +01:00
6ca7123d5d
Add comment for usage of YoutubeParsingHelper.resetClientVersionAndKey
2021-01-14 20:01:52 +01:00
11dcfe638b
Add durationPerFrame to Frameset and getFrameBoundsAt method
2021-01-14 20:01:06 +01:00
c2ff6723d7
Merge pull request #511 from B0pol/errors
...
Fix PeerTube tests by changing instance
2021-01-14 19:15:22 +01:00
6773ec8699
Merge pull request #510 from B0pol/heartedbycreator
...
comments: add heartedByUploader and extract it for YouTube
2021-01-12 19:00:04 +01:00
0c4836c800
[media.ccc.de] Recent kiosk: Add comment clarifying "length" and "duration" fields have the same value
2021-01-12 18:48:42 +01:00
4af50c6870
Use @link for downloader
2021-01-12 09:41:57 +01:00
3455f0f23c
Add documentation to testing with mocks
2021-01-12 09:37:48 +01:00
5292cc1636
[peertube] test: remove usage of peertube.mastodon.host
...
It's down for months
2021-01-11 19:46:27 +01:00
f3c22da618
comments: add heartedByCreator and extract it for youtube
2021-01-11 18:48:08 +01:00
63c237db41
Only create directories if they do not exist
2021-01-11 07:57:11 +01:00
e82cb1efea
Remove dependency to commons-io
2021-01-10 20:50:38 +01:00
1ea6c6ce54
Prefix mock file from RecordingDownloader with "generated_mock_"
...
Only read those files in MockDownloader
2021-01-10 20:46:42 +01:00
TiA4f8R
Fynn Godau
TobiGr
bopol
golfinq
XiangRongLin
FireMasterK
Stypox
vkay94