8806fb4e6b
address reviews
2021-03-19 10:14:43 +01:00
c47cc54908
Extract metadata for YouTube, SoundCloud & MediaCCC
2021-03-19 10:14:43 +01:00
379d7312fa
Don't use a regular expression to find the last segment URL and do code improvements
...
Apply suggestions provided in the PR and remove a redundant import.
2021-03-14 17:54:22 +01:00
0e3e420a25
Fix tests
2021-03-14 17:54:21 +01:00
d61d9d116d
Refactor getAudioStreams method of SoundcloudStreamExtractor
...
Split the method into private methods, in order to have a better reading.
2021-03-14 17:54:20 +01:00
a7b15b51e6
Change t to transcodingJsonObject in SoundcloudStreamExtractor
2021-03-14 17:54:19 +01:00
0438828e36
Add a test for the number of audioStreams in CreativeCommonsPlaysWellWithOthers test
...
It should be only two audio streams for track "Plays Well with Others, Ep 2: What Do an Army of Ants and an Online Encyclopedia Have in Common?" by Creative Commons (https://soundcloud.com/wearecc/plays-well-with-others-ep-2-what-do-an-army-of-ants-and-an-online-encyclopedia-have-in-common ):
- one which is a progressive stream, in MP3 format with a bitrate of 128 kbps
- one which is an HLS stream, in OPUS format with a bitrate of 64 kbps.
2021-03-14 17:54:18 +01:00
3bd08a2880
Adress requested changes and use final where possible in SoundcloudStreamExtractor
...
This commit moved the HLS parsing task to a separate method, did little performance improvements and used final where possible in the SoundcloudStreamExtractor file.
2021-03-14 17:54:17 +01:00
cbacd3c0a5
Add a check to don't show MP3 128kbps stream twice and catch IOException when fetching the HLS Manifest
...
If a progressive stream is present in the transcodings, it's unnecessary to show twice an MP3 128kbps stream so if this is the case, the MP3 HLS stream will be not added to the audioStreams, else it will.
This commit also catch fetching errors in HLS manifests parsing and don't add the corresponding stream if an error occurs.
2021-03-14 17:54:16 +01:00
26f1b4e7dc
Support SoundCloud HLS by using a workaround
...
This commit tries to support SoundCloud HLS streams by parsing M3U manifests, get the last segment URL (in order to get track length) and request a segment URL equals to track's duration so it's a single URL.
2021-03-14 17:54:14 +01:00
2e57a8f24f
[Bandcamp] Fix link handler acceptance behaviour
...
* Test for bandcamp footer instead of meta tag (which is not present on
all pages)
* Accept links to /music, not just /releases
* Correctly handle uppercase URLs
2021-03-14 09:51:30 +01:00
70d9e389b9
[Bandcamp] Add tests for external bandcamp artists
...
Necessary, because the external pages tested before were converted to bandcamp.com pages. See f9d06252f2
2021-03-13 20:51:59 +01:00
f9d06252f2
[Bandcamp] Fix tests
2021-03-13 17:42:39 +01:00
7e6f464407
Merge pull request #509 from TiA4f8R/soundcloud-improvements
...
Add new exceptions to be able to display different error messages in apps
2021-03-07 13:51:15 +01:00
ac51134aed
Merge pull request #546 from XiangRongLin/unignore2
...
Unignore more tests
2021-03-07 13:15:24 +01:00
22fa131922
Merge branch 'dev' into bandcamp
2021-03-05 19:45:37 +01:00
e55284bb8f
Code style improvements
...
Use final where possible in YoutubeStreamExtractor and do some other code style improvements
2021-03-05 16:38:53 +01:00
d0a1041afd
Prevent NullPointerException and use equalsIgnorecase
2021-03-05 16:38:52 +01:00
890cbba625
Update PaidContent exception
2021-03-05 16:38:51 +01:00
da3cfa967d
Handle age-restricted videos
2021-03-05 16:38:50 +01:00
448b68700c
Use final in exceptions and add AgeRestrictedException
...
AgeRestrictedException will be thrown only if the reason message equals to "Sign in to confirm your age" and if the age limit is 18.
2021-03-05 16:38:49 +01:00
22e6f33f0a
Ignore SoundCloudGeoRestrictedTest until HLS stream extraction is added
...
The ContentNotSupportedException is thrown because no supported audio streams where extracted. However, SoundCLoud does not check, whether there are any streams available.
This commit should be reverted in #526
2021-03-05 16:38:47 +01:00
21158744b1
Add expectedDescriptionIsEmpty() to DefaultStreamExtractorTest
...
Also check if related streams are empty if they are expected to be empty.
2021-03-05 16:38:46 +01:00
f15d7837a1
Fix SoundCloud tests
2021-03-05 16:38:45 +01:00
4e45aef2b3
Fix SoundCloud CreativeCommonsPlaysWellWithOthers test
2021-03-05 16:38:43 +01:00
59d6d3f04e
Fix YouTube tests
2021-03-05 16:38:42 +01:00
3925204658
Add tests and rename SoundcloudGoPlusException to SoundcloudGoPlusContentException
2021-03-05 16:38:40 +01:00
771bb1a2cb
Add three new exceptions for unavailable YouTube contents
...
These exceptions are thrown on a test with the error messages text, because YouTube returns only "UNPLAYABLE" status in most error cases.
Tests are based with English strings, so changing the lang used by
extractor will throw the generic exception (ContentNotAvailableException).
2021-03-05 16:38:39 +01:00
35325d980d
Add GeographicRestrictionException and SoundCloudGoPlusException in NewPipe Extractor to be able to display different error messages
...
This commit adds two new exceptions in NewPipe Extractor: GeographicRestrictionException and SoundCloudGoPlusException (which extend to ContentNotAvailableException). These exceptions allow showing different error messages to user when a content isn't available in his/her/its country (only used for now by SoundCloudStreamExtractor) or when the content is a SoundCloud Go+ track.
2021-03-05 16:38:37 +01:00
bc0cda68d1
Merge pull request #570 from TiA4f8R/fix-yt-channels-playlists
...
Fix YouTube channels contination
2021-03-05 15:26:49 +01:00
a3c6fceef5
Merge pull request #573 from B0pol/comments-performance
...
[youtube] improve comments extraction performance
2021-03-05 14:46:27 +01:00
ff5273b882
Update extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/extractors/YoutubeCommentsExtractor.java
...
Co-authored-by: Tobi <TobiGr@users.noreply.github.com>
2021-03-05 14:39:01 +01:00
ec1127dd6a
Merge pull request #571 from TeamNewPipe/soundcloud_client_id
...
[SoundCloud] Update client_id and fix extraction
2021-03-05 14:21:14 +01:00
ed850d0688
[youtube] improve comments extraction performance
...
- do not parse responseBody twice for continuation
instead try to get commentsTokenInside with the new pattern ("sectionListRenderer")
and try again with the old pattern ("commentSectionRenderer") on failure
- do not unescape responseBody multiple times
-> parse responseBody less times
2021-03-05 13:37:08 +01:00
440a808b8a
Update mocks for channels
2021-03-05 13:13:36 +01:00
05213175cd
minor edit, made variable final
2021-03-05 02:49:15 -05:00
8bceb57a6c
formatting fix
2021-03-05 02:46:06 -05:00
ae48bdea4c
Updated to reflect variable name change
2021-03-05 02:40:36 -05:00
b27efdc9a4
Remove clickTrackingParams in getNextPage method of YoutubeChannelExtractor
...
Because it's unused with the new request.
Co-authored-by: bopol <bopol@e.email>
2021-03-04 22:50:20 +01:00
2b671b15ce
[YouTube] Use new continuation API for channels
...
Co-authored-by: TiA4f8R <74829229+tia4f8r@users.noreply.github.com>
2021-03-04 21:47:16 +01:00
33594d4aff
[SoundCloud] Fix client id extraction
...
SoundCloud changed their Javascript structure. That caused the client_id to be in the middle of the file (at byte ~43000 ). To have a little buffer, we now fetch the first 50000 bytes.
2021-03-04 21:32:34 +01:00
fb2f37d223
[SoundCloud] Update hardcoded client id
2021-03-04 21:25:41 +01:00
506cc5fbe9
regenerate mocks
2021-03-04 19:01:58 +01:00
03b00ff1d6
Move toJsonArray and toJsonObject to JsonUtils
2021-03-04 18:58:51 +01:00
668b080b3a
Use dynamic key.
2021-03-04 08:45:14 +01:00
43b46bd408
Use new youtube continuations api for playlists
...
Requires sending a POST request instead of GET.
clientName and clientVersion, which were required as headers previously now need to be part of the request payload.
continuation id also needs to be part of request body.
quick and dirty solution.
2021-03-03 19:49:26 +01:00
beb05bd05c
Replace test url, because playlist was deleted
2021-03-03 19:46:14 +01:00
e13e237392
Fix typo and reword some explanations
2021-02-26 17:48:03 +01:00
a1688fe953
Move BandcampExtractorHelper.getJsonData(String, String) to JsonUtils
2021-02-21 13:51:12 +01:00
70814dcfef
Fix Utils.nonEmptyAndNullJoin
...
When using the index here, it the index needs to be decremented once an element is removed. To cirecumvent this, the native Collections.removeIf() method is used.
2021-02-21 12:51:47 +01:00
b9e8ee8450
Rename BandcampExtractorHelper.smartConcatenate(String[], String) to Utils.nonEmptyAndNullJoin(String, String[])
2021-02-21 12:35:14 +01:00
c07db80ef0
Add BASE_URL and BASE_API_URL to BandcampExtractorHelper
2021-02-21 12:15:45 +01:00
adde4332d1
Code improvements
2021-02-20 23:58:54 +01:00
42bdd8adc5
Update to fix failing mock tests.
2021-02-20 17:14:41 +05:30
e8bccfaf5d
Change UA to privacy.resistFingerprinting.
2021-02-20 17:14:41 +05:30
0c831afa7b
use DownloaderFactory instead of DownloaderTestImpl for search ChannelVerified
2021-02-20 11:41:30 +01:00
ea120a4637
remove print stacktrace
...
use interface
2021-02-20 00:23:45 +01:00
adf9d7d10f
Add reason field to MockOnly
...
This enforces developers to document why a test is skipped
2021-02-19 18:54:18 +01:00
ea52030613
Add MockOnlyRule to allow skipping specific tests based on downloader
2021-02-19 18:54:17 +01:00
02920fafa8
Add isUploaderVerified()
2021-02-19 15:51:23 +01:00
91e9309486
Merge remote-tracking branch 'origin/dev' into bandcamp
2021-02-19 13:37:45 +01:00
fa61b864f2
Code improvements
2021-02-19 13:37:24 +01:00
98268e351c
Move radio URL check into a function
2021-02-19 13:35:44 +01:00
5333d8a98b
Add more data.
2021-02-19 00:55:03 +05:30
1a322ad8ed
Add uploader verified by service extraction
2021-02-18 16:36:40 +01:00
c24afa2cbb
Permalink URL.
2021-02-18 20:16:53 +05:30
8d54401233
Add fps field as well.
2021-02-18 20:16:53 +05:30
5b70645d85
Add data to muxed streams aswell.
2021-02-18 20:16:53 +05:30
e975d33fbe
Change jdoc / comments.
2021-02-18 20:16:53 +05:30
f31b2a68fd
use getters/setters.
2021-02-18 20:16:53 +05:30
d4945ac55c
Constructor cleanup.
2021-02-18 20:16:52 +05:30
3f6a601be8
Fix jdoc / comments.
2021-02-18 20:16:52 +05:30
a7c9905183
Fix requested changes.
2021-02-18 20:16:52 +05:30
0c0f2d74bc
Make fields private
...
Co-authored-by: bopol <bopol@e.email>
2021-02-18 20:16:52 +05:30
d1054338d3
Fix error with livestreams.
2021-02-18 20:16:52 +05:30
11eb4932f4
Add data to respective classes.
2021-02-18 20:16:52 +05:30
525e345ed8
remove duplicate field.
2021-02-18 20:16:52 +05:30
e844d2aed3
Add dash fields to the ItagItem.
2021-02-18 20:16:52 +05:30
64f6b0478f
Merge pull request #545 from XiangRongLin/comment_reset
...
Add comment explaining YoutubeParsingHelper.resetClientVersionAndKey
2021-02-18 15:14:03 +01:00
a64579da2f
Unignore some tests that work on Linux
2021-02-17 20:31:26 +01:00
b99cb8624f
Unignore PublicBroadcasterTest
...
Tests only work on linux, mocks have to be generated on linux too
2021-02-17 20:31:26 +01:00
37366c65ca
Remove encoding of '&' to '&' in test and unignore it
2021-02-17 20:31:26 +01:00
c327922cc0
Add explanations to broken tests
2021-02-17 20:31:26 +01:00
26d50aca05
Add comment explaining YoutubeParsingHelper.resetClientVersionAndKey
2021-02-17 19:21:39 +01:00
54aa5b3042
Use propper structure in KioskExtractors
...
Made BandCampRadioExtractor a Kiosk which holds StreamInfoItems and not InfoItems.
2021-02-16 21:09:42 +01:00
54b8e54f80
Fix potential NPE
2021-02-16 20:59:13 +01:00
16973126a0
Use Collections.emptyList() instead of Collections.EMPTY_LIST or new ArrayList()
2021-02-16 20:25:48 +01:00
84380e4c3a
Merge pull request #536 from B0pol/watchendpoints
...
support /watch/, /v/ and /w/ subpaths
2021-02-16 20:23:27 +01:00
41f689b099
Update extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/linkHandler/YoutubeStreamLinkHandlerFactory.java
...
Co-authored-by: Stypox <stypox@pm.me>
2021-02-16 19:17:10 +01:00
a9594f531f
Merge pull request #543 from XiangRongLin/unignore1
...
Unignore and fix some tests
2021-02-16 18:00:16 +01:00
a28e0226c5
Merge pull request #519 from XiangRongLin/better_logging_error_handling
...
Better logging, error handling, doc for mock tests
2021-02-16 15:06:06 +01:00
05cb22c654
Unignore Gronkh.testDescription
...
Remove special character ü from assertion
2021-02-16 08:59:35 +01:00
d9b4c2779f
Unignore YoutubeSearchQHTest.testRegularValues
2021-02-16 08:59:35 +01:00
137272b7a4
Unignore YoutubeCommentsExtractorTest.EmptyComment
...
Mock files were missing
2021-02-16 08:59:34 +01:00
5bf9fddba9
Code improvements
2021-02-15 22:45:17 +01:00
e062c8cb0d
Merge branch 'dev' into bandcamp
2021-02-15 22:09:41 +01:00
7a3d9bdb7d
Description: add .equals() and .hashCode()
2021-02-15 18:51:47 +01:00
557934cb17
use EMPTY_STRING
2021-02-15 18:51:46 +01:00
9ca52ca68a
use only one constant UTF-8
2021-02-15 18:47:54 +01:00
50e5718bd1
Merge pull request #518 from XiangRongLin/generate_mocks
...
Use and generate mocks for youtube tests
2021-02-15 06:39:41 +01:00
bopol
TiA4f8R
Fynn Godau
TobiGr
golfinq
XiangRongLin
FireMasterK