ed850d0688
[youtube] improve comments extraction performance
...
- do not parse responseBody twice for continuation
instead try to get commentsTokenInside with the new pattern ("sectionListRenderer")
and try again with the old pattern ("commentSectionRenderer") on failure
- do not unescape responseBody multiple times
-> parse responseBody less times
2021-03-05 13:37:08 +01:00
440a808b8a
Update mocks for channels
2021-03-05 13:13:36 +01:00
b608587e4d
Merge pull request #572 from golfinq/fixed-comments
...
Fixes missing comments under videos
2021-03-05 11:42:40 +01:00
05213175cd
minor edit, made variable final
2021-03-05 02:49:15 -05:00
8bceb57a6c
formatting fix
2021-03-05 02:46:06 -05:00
ae48bdea4c
Updated to reflect variable name change
2021-03-05 02:40:36 -05:00
b27efdc9a4
Remove clickTrackingParams in getNextPage method of YoutubeChannelExtractor
...
Because it's unused with the new request.
Co-authored-by: bopol <bopol@e.email>
2021-03-04 22:50:20 +01:00
2b671b15ce
[YouTube] Use new continuation API for channels
...
Co-authored-by: TiA4f8R <74829229+tia4f8r@users.noreply.github.com>
2021-03-04 21:47:16 +01:00
33594d4aff
[SoundCloud] Fix client id extraction
...
SoundCloud changed their Javascript structure. That caused the client_id to be in the middle of the file (at byte ~43000 ). To have a little buffer, we now fetch the first 50000 bytes.
2021-03-04 21:32:34 +01:00
fb2f37d223
[SoundCloud] Update hardcoded client id
2021-03-04 21:25:41 +01:00
9256b3b848
Merge pull request #567 from XiangRongLin/playlist_continuations
...
Playlist continuations
2021-03-04 19:05:32 +01:00
506cc5fbe9
regenerate mocks
2021-03-04 19:01:58 +01:00
03b00ff1d6
Move toJsonArray and toJsonObject to JsonUtils
2021-03-04 18:58:51 +01:00
668b080b3a
Use dynamic key.
2021-03-04 08:45:14 +01:00
43b46bd408
Use new youtube continuations api for playlists
...
Requires sending a POST request instead of GET.
clientName and clientVersion, which were required as headers previously now need to be part of the request payload.
continuation id also needs to be part of request body.
quick and dirty solution.
2021-03-03 19:49:26 +01:00
beb05bd05c
Replace test url, because playlist was deleted
2021-03-03 19:46:14 +01:00
e13e237392
Fix typo and reword some explanations
2021-02-26 17:48:03 +01:00
5d594cfded
Merge pull request #549 from FireMasterK/gradle-update
...
Update gradle wrapper.
2021-02-24 15:59:28 +01:00
965a66bc87
Merge pull request #557 from TiA4f8R/fix-encoding-errors-windows
...
Set extractor encoding to UTF-8
2021-02-24 15:49:58 +01:00
b2cf41496d
Set extractor encoding to UTF-8
...
When compiling the extractor on Windows, you get more than 100 errors with the message error: unmappable character for encoding Cp1252, especially in timeago-parser tasks. This commit fixes it by setting the encoding for all compileJava and compileTestJava tasks to UTF-8.
2021-02-23 14:38:20 +01:00
a1688fe953
Move BandcampExtractorHelper.getJsonData(String, String) to JsonUtils
2021-02-21 13:51:12 +01:00
70814dcfef
Fix Utils.nonEmptyAndNullJoin
...
When using the index here, it the index needs to be decremented once an element is removed. To cirecumvent this, the native Collections.removeIf() method is used.
2021-02-21 12:51:47 +01:00
b9e8ee8450
Rename BandcampExtractorHelper.smartConcatenate(String[], String) to Utils.nonEmptyAndNullJoin(String, String[])
2021-02-21 12:35:14 +01:00
c07db80ef0
Add BASE_URL and BASE_API_URL to BandcampExtractorHelper
2021-02-21 12:15:45 +01:00
adde4332d1
Code improvements
2021-02-20 23:58:54 +01:00
42bdd8adc5
Update to fix failing mock tests.
2021-02-20 17:14:41 +05:30
e8bccfaf5d
Change UA to privacy.resistFingerprinting.
2021-02-20 17:14:41 +05:30
a9aa385bb3
Merge pull request #554 from B0pol/test-verified
...
use DownloaderFactory instead of DownloaderTestImpl for search Channe…
2021-02-20 12:12:43 +01:00
0c831afa7b
use DownloaderFactory instead of DownloaderTestImpl for search ChannelVerified
2021-02-20 11:41:30 +01:00
ea120a4637
remove print stacktrace
...
use interface
2021-02-20 00:23:45 +01:00
adf9d7d10f
Add reason field to MockOnly
...
This enforces developers to document why a test is skipped
2021-02-19 18:54:18 +01:00
ea52030613
Add MockOnlyRule to allow skipping specific tests based on downloader
2021-02-19 18:54:17 +01:00
02920fafa8
Add isUploaderVerified()
2021-02-19 15:51:23 +01:00
91e9309486
Merge remote-tracking branch 'origin/dev' into bandcamp
2021-02-19 13:37:45 +01:00
fa61b864f2
Code improvements
2021-02-19 13:37:24 +01:00
98268e351c
Move radio URL check into a function
2021-02-19 13:35:44 +01:00
27a20e41cd
Merge pull request #550 from FireMasterK/add-more-data
...
Add more data.
2021-02-18 20:28:27 +01:00
5333d8a98b
Add more data.
2021-02-19 00:55:03 +05:30
88890d81a5
Update gradle wrapper.
2021-02-18 21:15:12 +05:30
9c1a7f7df6
Merge pull request #521 from B0pol/verified
...
Add uploader verified by service extraction
2021-02-18 16:39:53 +01:00
1a322ad8ed
Add uploader verified by service extraction
2021-02-18 16:36:40 +01:00
db253e202b
Merge pull request #535 from TeamPiped/adaptive-parsing
...
Adaptive Stream parsing
2021-02-18 15:55:27 +01:00
c24afa2cbb
Permalink URL.
2021-02-18 20:16:53 +05:30
8d54401233
Add fps field as well.
2021-02-18 20:16:53 +05:30
4d096be14a
Revert "Update gradle wrapper."
...
This reverts commit 591856e3a11998c4b8ba87b5e361e2b9556d760f.
2021-02-18 20:16:53 +05:30
5b70645d85
Add data to muxed streams aswell.
2021-02-18 20:16:53 +05:30
e975d33fbe
Change jdoc / comments.
2021-02-18 20:16:53 +05:30
f31b2a68fd
use getters/setters.
2021-02-18 20:16:53 +05:30
d4945ac55c
Constructor cleanup.
2021-02-18 20:16:52 +05:30
3f6a601be8
Fix jdoc / comments.
2021-02-18 20:16:52 +05:30
bopol
TobiGr
golfinq
TiA4f8R
XiangRongLin
FireMasterK