24eba62305
fix: extract YouTube trends from new "Videos" tab
2023-05-03 21:16:23 +02:00
a9ca5c49e4
Merge pull request #1056 from AudricV/yt-improve-search-suggestions-extraction
...
[YouTube] Switch to new search suggestion domain and improve error handling
2023-05-02 20:17:48 +01:00
108f8a7a17
Bump org.jsoup:jsoup from 1.15.4 to 1.16.1
...
Bumps [org.jsoup:jsoup](https://github.com/jhy/jsoup ) from 1.15.4 to 1.16.1.
- [Release notes](https://github.com/jhy/jsoup/releases )
- [Changelog](https://github.com/jhy/jsoup/blob/master/CHANGES )
- [Commits](https://github.com/jhy/jsoup/compare/jsoup-1.15.4...jsoup-1.16.1 )
---
updated-dependencies:
- dependency-name: org.jsoup:jsoup
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2023-05-01 09:58:32 +00:00
cf6040ddb3
Update YoutubeSuggestionExtractorTest mocks
2023-04-30 19:53:44 +02:00
593122342f
[YouTube] Improve YoutubeSuggestionExtractorTest
...
- Remove useless concatenation on the downloader path;
- Remove unneeded public test modifier;
- Update license header;
- Specify the service class tested instead of the generic class.
2023-04-30 19:53:43 +02:00
e923fca440
[YouTube] Switch to new search suggestion domain and improve error handling
...
- Switch to the new domain used by YouTube for search suggestions,
suggestqueries-clients6.youtube.com, and add the xhr query parameter with the
t value, to allow getting responses without requiring trim;
- Use the Java 8 Stream API to collect search suggestions and improve invalid
response detection by checking whether the content type of the response
returned is JSON;
- Move the licence header at the top of the file.
2023-04-30 19:53:42 +02:00
945165a3c0
[PeerTube] Don't return "No description" when there is no description for a channel or an account
...
When a description is missing, no description should be returned, even the ones
indicating there is no description. This behavior is represented by a null
return instead.
Also update PeertubeAccountExtractorTest to reflect these changes.
2023-04-30 18:41:38 +02:00
2deb023da4
Merge pull request #1050 from Theta-Dev/fix/channel-carousel-header
...
[YouTube] Add support for CarouselHeaderRenderer
2023-04-25 15:17:31 +02:00
4aada7f91b
refactor: rename carousel header channel test
2023-04-21 22:48:37 +02:00
47aa9fed40
fix: set musicClientVersion regex capture group
2023-04-16 19:25:05 +02:00
20370395c5
fix: add support for CarouselHeaderRenderer
2023-04-16 17:40:13 +02:00
7dba6e3891
Merge pull request #1033 from petlyh/bandcamp-paywalled-content
...
[Bandcamp] Handle paywalled tracks
2023-04-12 13:04:26 +02:00
e6aad117e7
[Bandcamp] Throw PaidContentException on paywalled albums
2023-04-03 19:27:09 +02:00
69705138e4
[Bandcamp] Fix extraction of related playlist items URL ( #1047 )
...
Small change in HTML structure
2023-04-02 22:24:29 +02:00
1b6fe5edd6
[YouTube] Fix ParsingException when comments are unavailable in a video ( #1040 )
...
Co-authored-by: bjs <bjs@elect-it.com>
Co-authored-by: Audric V. <74829229+AudricV@users.noreply.github.com>
Co-authored-by: Kavin <20838718+FireMasterK@users.noreply.github.com>
2023-03-30 19:58:06 +02:00
8d1303e18f
Add track types to audio streams ( #1041 )
2023-03-28 00:02:20 +02:00
80a6fc2c63
[PeerTube] Fix testGetCommentsFromCommentsInfo test of PeertubeCommentsExtractorTest.Default
...
The tested comment has been removed, so it couldn't be found in the comments
list.
This comment has been replaced by a new one from the current comments of the
video.
Also, in the parent class PeertubeCommentsExtractorTest, final has been used as
much as possible and for-each loops of lists have been replaced by their
forEach method or the Stream API, in order to simplify code.
2023-03-04 16:49:10 +01:00
5a9b6ed2e3
[Bandcamp] Support loading additional comments ( #1030 )
2023-03-04 14:01:06 +01:00
6bdd698c25
Merge pull request #1026 from AudricV/audio-streams-descriptive-and-locale-properties
...
Add descriptive and locale properties to audio streams
2023-03-01 11:15:46 +01:00
19e4b216c9
Merge pull request #1032 from AudricV/yt_fix-comments-hashtags-links-extraction
...
[YouTube] Fix hashtags links extraction and escape HTML links
2023-03-01 10:47:37 +01:00
b1298490c0
Merge pull request #1029 from AudricV/yt_fix-no-views-extraction-playlist-items
...
[YouTube] Fix partial non-extraction of "No views" string in stream items
2023-03-01 10:46:52 +01:00
9dc1832733
[Bandcamp] Handle paywalled tracks
2023-02-28 17:51:30 +01:00
3fdb6ee476
Merge pull request #1031 from petlyh/bandcamp-fix-radio-comments
...
[Bandcamp] Show comments as disabled on radio streams
2023-02-27 20:50:12 +01:00
bd79b921e8
[YouTube] Refactor the code to get stream items' view count
...
This refactoring avoids code duplication as much as possible.
2023-02-27 10:25:46 +01:00
51f9b39953
[YouTube] Fix partial non-extraction of no views string in stream items
...
As the "No views" string is returned in the case there is no view on a video, a
number cannot be parsed in this case, so -1 was returned.
This string is now detected in all methods to get the view count of a stream.
2023-02-27 10:18:45 +01:00
95b3f5e391
[MediaCCC] Test audio language property extraction
2023-02-26 19:06:18 +01:00
30a0f8c510
[MediaCCC] Extract audio language property for single language audio tracks
2023-02-26 19:06:18 +01:00
7f0269c4c7
[YouTube] Edit YoutubeStreamExtractorDefaultTest.AudioTrackLanguage to test audio locale property
...
The Hindi audio track language presence test has been changed from audio track
label to audio locale.
2023-02-26 19:06:18 +01:00
034f82dae7
[YouTube] Test language and descriptive audio in YoutubeDashManifestCreatorsTest
2023-02-26 19:06:17 +01:00
05e8cb39f7
[YouTube] Add language and descriptive audio properties to DASH manifests
2023-02-26 19:06:17 +01:00
bf30d70152
[YouTube] Add descriptive audio test
...
This test uses video TjxC-evzxdk.
Also improve a bit YoutubeStreamExtractorDefaultTest.AudioTrackLanguage test.
2023-02-26 19:06:17 +01:00
76b7c19c5d
[YouTube] Extract whether a track is a descriptive audio and audio locale when available
...
Also use audio track setters only for audio itags.
2023-02-26 19:06:17 +01:00
3bb5eeef30
[YouTube] Add descriptive and locale audio support in ItagItem
2023-02-26 19:06:16 +01:00
14bf3fb05b
Add ability to know the locale of an audio stream
...
Getting audio tracks locales by parsing their ID or their label, should not be
done by clients, but by the extractor.
This commit adds the ability to store the Locale of an AudioStream, which is
used to compare similar AudioStreams (in the equalStats method).
2023-02-26 19:06:16 +01:00
f92426560c
Add descriptive audio properties
...
Also improve AudioStream's audio language documentation
2023-02-26 19:06:16 +01:00
a63f289667
[YouTube] Update mocks of YoutubeCommentsExtractorTest.FormattingTest
2023-02-26 18:50:07 +01:00
9483dcd9fa
[YouTube] Update mocks of YoutubeCommentsExtractorTest.RepliesTest
2023-02-26 18:43:36 +01:00
1556adbb2d
[YouTube] Fix hashtags links extraction and escape text in attribute descriptions + HTML links
...
webCommandMetadata object is contained inside a commandMetadata one, so it is
not accessible from the root of the navigationEndpoint object.
The corresponding statement has been moved at the bottom of the specific
endpoints parsing, as the webCommandMetadata object is present almost
everywhere, otherwise URLs of some endpoints would have be changed, such as
uploader URLs (from channel IDs to handles).
As no ParsingException is now thrown by getUrlFromNavigationEndpoint, and so by
getTextFromObject, getUrlFromObject and getTextAtKey, the methods which were
catching ParsingExceptions thrown by these methods had to be updated.
URLs got in the HTML version of getTextFromObject are now escaped properly to
provide valid HTML to clients. This has been also done for attribute
descriptions, with the description text for this type of descriptions.
As YouTube descriptions are in HTML format (except for the fallback on the JSON
player response, which is plain text and only happens when there is no visual
metadata or a breaking change), all URLs returned are escaped, so tests which
are testing presence of URLs with escaped characters had to be updated (it was
only the case for YoutubeStreamExtractorDefaultTest.DescriptionTestUnboxing).
2023-02-26 18:43:36 +01:00
f7a7a236fb
[Bandcamp] Show comments as disabled on radio streams
2023-02-23 18:42:43 +01:00
f5599ff08d
Bump org.jsoup:jsoup from 1.15.3 to 1.15.4
...
Bumps [org.jsoup:jsoup](https://github.com/jhy/jsoup ) from 1.15.3 to 1.15.4.
- [Release notes](https://github.com/jhy/jsoup/releases )
- [Changelog](https://github.com/jhy/jsoup/blob/master/CHANGES )
- [Commits](https://github.com/jhy/jsoup/compare/jsoup-1.15.3...jsoup-1.15.4 )
---
updated-dependencies:
- dependency-name: org.jsoup:jsoup
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2023-02-20 10:05:26 +00:00
3f7df9536e
[YouTube] Fix getting the comment text if the comment contains a hashtag
2023-01-29 20:33:51 +01:00
999fb7f812
Merge pull request #1024 from AudricV/snd_fix-tracks-like-count
...
[SoundCloud] Fix extraction of tracks like count
2023-01-29 10:52:54 +01:00
3519d4c367
Merge pull request #1015 from AudricV/yt_fix-channel-id-rss-feeds
...
[YouTube] Fix channel ID extraction of YouTube channels RSS feeds
2023-01-29 10:41:38 +01:00
9aca710e86
Merge pull request #1013 from Stypox/fix-music-mixes
...
[YouTube] Now music mixes can be treated as normal mixes
2023-01-29 09:48:51 +01:00
76eeabac45
Merge pull request #1020 from TeamNewPipe/fix/yt-subscriber-count
...
[YouTube] Fix NPE in search when getting channel items without subscriber count
2023-01-29 09:44:22 +01:00
676622f6df
[SoundCloud] Fix expectedLikeCountAtLeast tests of SoundcloudStreamExtractorTest test classes
...
As like count is now returned by the extractor, we need to assert a positive
minimum like count, which is close to the actual value, in order to avoid test
failures due to lower like counts than the ones excepted.
2023-01-29 01:08:02 +01:00
2a24d407d5
[SoundCloud] Fix extraction of tracks like count
...
SoundCloud is using likes_count to return the like count of a track, like it
was the case before they switched to favoritings_count.
2023-01-29 01:00:49 +01:00
ba24976e41
[YouTube] Add live URLs test and do minor improvements to YoutubeStreamLinkHandlerFactoryTest
...
- Remove unused imports;
- Replace wildcard imports by single class imports;
- Suppress "HTTP links are not secured" warnings from IDEA IDEs;
- Replace removed video jZViOEv90dI by an existing video, 9Dpqou5cI08 (the
corresponding test has been of course renamed).
2023-01-28 19:36:21 +01:00
57f850bc2d
[YouTube] Support live URLs and do minor improvements to YoutubeStreamLinkHandlerFactory
...
- Move license header at the top;
- Use an unmodifiable set for the subpaths instead of a modifiable list;
- Add missing Nonnull and Nullable annotations;
- Improve exception messages.
2023-01-28 19:36:20 +01:00
1f4ed9dce9
[YouTube] Fix channel ID extraction of YouTube channel RSS feeds
...
The yt:channelId element doesn't provide the channel ID anymore and is empty,
like the id element, so we need now to extract it from the channel URL provided
in two elements: author -> uri and feed -> link.
Also avoid a NullPointerException in getUrl and getName methods.
2023-01-28 11:53:33 +01:00
c589a2c1a2
Merge pull request #1014 from TeamNewPipe/fix/yt-comments
...
[YouTube] Fix getting next comments pages
2023-01-27 11:14:55 +01:00
72573932cf
[YouTube] Fix NPE in search when getting channel items without subscriber count
2023-01-24 23:03:45 +01:00
f50b7275af
[YouTube] Fix getting next comments pages
2023-01-24 22:39:08 +01:00
9bdad40b06
Removed topStandaloneBadge
2023-01-20 02:41:21 +05:30
5945057227
[YouTube] Add music mix test
2023-01-15 23:30:30 +01:00
7293991832
[YouTube] Now music mixes can be treated as normal mixes
...
Using a playlist extractor on them would result in "Unviewable playlist" errors
2023-01-15 23:28:59 +01:00
ff94e9f30b
Merge pull request #1009 from TeamNewPipe/dependabot/gradle/com.google.code.gson-gson-2.10.1
...
Bump gson from 2.10 to 2.10.1
2023-01-11 15:35:36 +01:00
c1040bccac
Merge pull request #794 from FireMasterK/comments-count
...
[YouTube] Add support to extract total comment count
2023-01-11 15:32:19 +01:00
f43049985e
Bump gson from 2.10 to 2.10.1
...
Bumps [gson](https://github.com/google/gson ) from 2.10 to 2.10.1.
- [Release notes](https://github.com/google/gson/releases )
- [Changelog](https://github.com/google/gson/blob/master/CHANGELOG.md )
- [Commits](https://github.com/google/gson/compare/gson-parent-2.10...gson-parent-2.10.1 )
---
updated-dependencies:
- dependency-name: com.google.code.gson:gson
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2023-01-09 09:05:55 +00:00
56aab4d971
[YouTube] Fix escaping links in YouTubeParsingHelper.getTextFromObject
2023-01-05 00:28:12 +01:00
22a47da8c7
Fix requested change and remove outdated comment.
2023-01-02 20:42:32 +00:00
98a90fd9c8
Don't cache comments count and return early on page fetch if no token.
2023-01-02 20:40:48 +00:00
2974dfaa48
Only store ajaxJson for initial page and eager fetch the initial continuation.
2023-01-02 20:40:48 +00:00
64d24aa09e
Fix request changes.
2023-01-02 20:40:48 +00:00
67ef4f4c30
Cleanup and remove optional.
2023-01-02 20:40:48 +00:00
22f71b010c
Fix for requested changes.
2023-01-02 20:40:48 +00:00
656b7c1cd9
Improve method documentation.
2023-01-02 20:40:48 +00:00
981aee4092
Add support to extract total comment count.
2023-01-02 20:40:48 +00:00
45636b0d00
Merge pull request #986 from Isira-Seneviratne/Static_maps
...
Use immutable Map factory methods.
2023-01-02 18:11:14 +01:00
219c5c5be5
Update extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/YoutubeParsingHelper.java
2023-01-02 18:11:03 +01:00
259de3cba6
Merge pull request #995 from TeamNewPipe/feat/soundcloud-playlistinfoitemextractor
...
[SoundCloud] Implement getUploaderUrl() and isUploaderVerified() for PlaylistInfoItemExtractor
2023-01-02 15:10:40 +01:00
991394b53a
Merge pull request #1005 from FireMasterK/fix-escaping-xss
...
Fix for potential XSS attacks and formatting issues
2023-01-02 15:06:17 +01:00
d8ce08d969
Use immutable Map factory methods.
2023-01-02 07:50:31 +05:30
01acf79436
Fix for potential XSS attacks.
2022-12-31 20:05:32 +00:00
292e0d8ce7
[SoundCloud] Implement getUploaderUrl() and isUploaderVerified() for PlaylistInfoItemExtractor
2022-12-31 18:46:39 +01:00
2a8729aeb2
Apply suggestions
...
Co-authored-by: Stypox <stypox@pm.me>
2022-12-31 18:24:33 +01:00
d75a997611
[PeerTube] Support searching for channels
2022-12-31 18:24:33 +01:00
dea6d8ce4c
[PeerTube] Support searching for playlists
2022-12-31 18:24:33 +01:00
95cc6aefbb
Merge pull request #994 from TeamNewPipe/fix/peertube-subtitles-exception
...
[PeerTube] Report Exceptions thrown while getting a stream's subtitles
2022-12-31 15:01:39 +01:00
7b54457789
Merge pull request #941 from TeamNewPipe/feat/peertube-comment-replies
...
[PeerTube] Support comment replies
2022-12-31 14:57:51 +01:00
f45966d449
Merge pull request #910 from Isira-Seneviratne/Locale_forLanguageTag
...
Add compat Locale.forLanguageTag() implementation.
2022-12-24 23:53:30 +01:00
d5437e0bc5
Merge pull request #863 from AudricV/add-content-type-and-content-length-headers-to-post-requests
...
Add Content-Type header to all POST requests without an empty body
2022-12-16 19:32:56 +01:00
0766b1d211
[YouTube] Improve YoutubeStreamInfoItemExtractor
...
- Return duration of video premieres;
- Add another non-localized method to determine whether a stream is a running
livestream;
- Return view count and upload date of videos in playlists;
- Store isPremiere result;
- Remove shorts workaround code, as it was only useful on channels and shorts
have been moved into a separated channel tab;
- Improve some other code.
2022-12-08 13:59:12 +01:00
896d7e09eb
Merge pull request #978 from Theta-Dev/fix/search-channel-handles
...
[YouTube] Fix search subscriber count extraction with channel handles
2022-12-05 17:52:05 +01:00
cd3262745d
[PeerTube] Report Exceptions thrown while getting a stream's subtitles
2022-12-03 16:11:21 +01:00
4e66b2287e
[PeerTube] Add support for comment replies
2022-12-01 14:05:18 +01:00
41c8dce452
Merge pull request #992 from Isira-Seneviratne/String_isBlank
...
Use String.isBlank().
2022-11-30 17:48:54 +01:00
2bca56f0df
Use String.isBlank().
2022-11-30 08:26:21 +05:30
3b80547976
Add code review suggestions.
2022-11-30 07:57:45 +05:30
016623131e
docs: update comment in YoutubeChannelInfoItemExtractor
2022-11-29 19:06:03 +01:00
2e08eaad96
Fix complication error in comment test.
2022-11-29 16:07:48 +00:00
abf08e1496
Merge pull request #990 from FireMasterK/bold-italic-strikethrough
...
[YouTube] Implement bold/italic/strike-through support
2022-11-29 15:59:38 +00:00
57e7a6fb7c
Add mocks test.
2022-11-28 20:27:55 +00:00
1d3d7fa5c3
Add test for formatting.
2022-11-28 20:26:37 +00:00
52fda37915
Implement bold/italic/strike-through support.
2022-11-28 19:06:18 +00:00
b566084cac
Use Description object for comments text.
2022-11-28 17:02:19 +00:00
f8162b049d
Merge pull request #984 from FireMasterK/unused-dep
...
Remove unused autolink dependency
2022-11-28 11:28:42 +01:00
1da0190056
Merge pull request #980 from TeamNewPipe/fix/yt/unavailable
...
[YouTube] Fix extracting the detailed error message for unavailable streams
2022-11-28 10:07:34 +01:00
60fb30f835
Merge pull request #928 from FireMasterK/comment-urls
...
Parse YouTube comments as HTML
2022-11-27 19:16:34 +01:00
5abea22225
Fix throwing correct reason.
2022-11-26 21:09:08 +00:00
faf28f5c11
Remove unused dependency.
2022-11-26 20:17:25 +00:00
c043597255
Update supported countries list.
2022-11-26 19:01:33 +00:00
4680df0bdf
Fix throwing correct reason
2022-11-23 17:03:22 +01:00
9de8405c9f
[YouTube] Fix extracting the detailed error message of streams which are unavailable
2022-11-23 08:33:06 +01:00
34d79bd267
[YouTube] Update mocks
2022-11-22 17:10:04 +01:00
2ec296e674
Fix YoutubeSearchExtractorTest.MetaInfoTest
...
Not all the "learn more" button is uppercase anymore, that's only the case for
the first letter.
2022-11-22 16:34:54 +01:00
3891542ca1
Use Downloader's postWithContentType and postWithContentTypeJson methods in services and extractors
2022-11-22 11:37:18 +01:00
b2862f3cd1
Add postWithContentType and postWithContentTypeJson utility methods in Downloader
...
Co-authored-by: Stypox <stypox@pm.me>
2022-11-22 11:37:17 +01:00
e9a0d3bd95
[YouTube] Send Content-Type header in all POST requests
...
This header was not sent partially before and was added and guessed by OkHttp. This can create issues when using other HTTP clients than OkHttp, such as Cronet.
Some code in the modified classes has been improved and / or deduplicated, and usages of the UTF_8 constant of the Utils class has been replaced by StandardCharsets.UTF_8 where possible.
Note that this header has been not added in except in YoutubeDashManifestCreatorsUtils, as an empty body is sent in the POST requests made by this class.
2022-11-22 11:37:16 +01:00
b9e463de49
[Bandcamp] Send Content-Type header in POST requests
...
This header was not sent before and was added and guessed by OkHttp. This can create issues when using other HTTP clients than OkHttp, such as Cronet.
Also make use of StandardCharsets.UTF_8 when getting bytes of bodies instead of the platform default's charset, to make sure to prevent some encoding issues on some JVMs.
2022-11-22 11:35:46 +01:00
65d6321e3d
Fix typos in Downloader.post JavaDocs
...
Post methods in Downloader return the result of a POST request and not the one of a GET request.
2022-11-22 11:35:46 +01:00
5daabd1793
fix : #976 search subscriber count extraction with channel handles
2022-11-22 02:17:10 +01:00
c953e23414
Merge pull request #968 from AudricV/yt-support-no-video-info-renderers-for-streams
...
[YouTube] Support lack of video info renderers for streams
2022-11-16 20:20:01 +00:00
2211a24b69
Merge pull request #971 from lrusso96/patch-1
...
[YouTube] Improve duration parsing
2022-11-16 16:14:54 +01:00
86f06b333a
Address review.
2022-11-14 00:05:31 +00:00
b16e6082e1
Add test for audio stream languages.
2022-11-13 23:10:44 +00:00
30909da1df
Fix audio track similar comparison for IDs.
2022-11-13 23:08:54 +00:00
6d59cdbe3a
Add support for extracting audio tracks.
2022-11-13 21:39:29 +00:00
e4d982c7ea
Fix license.
2022-11-12 07:29:15 +05:30
416089146e
Fix failing tests.
2022-11-12 07:29:15 +05:30
ddbce3b83d
Add Utils methods for URL encoding/decoding using UTF-8.
2022-11-12 07:29:15 +05:30
366f5c1632
Use StandardCharsets.UTF_8.
2022-11-12 07:29:15 +05:30
0b82fade51
Bump gson from 2.9.1 to 2.10
...
Bumps [gson](https://github.com/google/gson ) from 2.9.1 to 2.10.
- [Release notes](https://github.com/google/gson/releases )
- [Changelog](https://github.com/google/gson/blob/master/CHANGELOG.md )
- [Commits](https://github.com/google/gson/compare/gson-parent-2.9.1...gson-parent-2.10 )
---
updated-dependencies:
- dependency-name: com.google.code.gson:gson
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2022-11-09 16:21:16 +00:00
c9635218e2
[YouTube] Improve duration parsing
2022-11-09 09:41:29 +01:00
316d8573fa
Use immutable sets in YoutubeParsingHelper.
2022-11-07 07:50:26 +05:30
6a2c680d8f
[YouTube] Add mocks for YoutubeStreamExtractorDefaultTest.NoVisualMetadataVideoTest
2022-11-04 19:43:06 +01:00
e66fed41d6
[YouTube] Add a StreamExtractor test for a video without visual metadata
...
The video "Makani’s first commercial-scale energy kite" (video ID:
An8vtD1FDqs), which has this behavior, is used for the new test,
NoVisualMetadataVideoTest, added in YoutubeStreamExtractorDefaultTest.
Tests of elements who throw an exception in this case (subscriber count, like
count, uploader avatar URL) test if the ParsingException exception is thrown by
YoutubeStreamExtractor.
2022-11-04 19:42:12 +01:00
aa9a8ca23c
[YouTube] Make non-extraction of videoPrimaryInfoRenderer and/or videoSecondaryInfoRenderer not fatal
...
Also de-duplicated common code related to the obtain of these video info renderers.
This change allows extraction of videos without visual metadata.
2022-11-04 18:35:53 +01:00
20cd8e8a4a
[YouTube] Update mocks of YoutubeChannelExtractorTest.Gronkh
2022-11-03 19:46:42 +01:00
a34f060ba0
[YouTube] Use a handle for YoutubeChannelExtractorTest.Gronkh
2022-11-03 19:46:42 +01:00
724f669ff7
[YouTube] Add tests for handles and user IDs with non ASCII characters support
...
Unneeded public modifiers in test methods of
YoutubeChannelLinkHandlerFactoryTest have been also removed.
2022-11-03 19:46:42 +01:00
61ce041bda
[YouTube] Support handles and all custom channel names
...
More non-channel paths have been also added to the excluded custom name paths,
documentation and exception messages have been improved and fixed in some
places, and the licence header of YoutubeChannelLinkHandlerFactory has been
moved to its beginning and updated.
2022-11-03 19:46:42 +01:00
ffffb04439
Merge pull request #953 from Theta-Dev/attributed-text-desc
...
[YouTube] Add support for attributed text description
2022-11-03 18:34:30 +01:00
592e1d6386
fix: parsing attributed description with no command runs
2022-11-03 12:10:52 +01:00
099b53cc4f
[YouTube] Add parser for attributedDescription
...
Also update the mock of the next InnerTube endpoint response of the
YoutubeStreamExtractorDefaultTest.DescriptionTestUnboxing test class with an
attributedDescription instead of a regular description
2022-11-02 23:11:33 +01:00
e4c24d4c36
[YouTube] Regenerate supported channels mocks
2022-11-02 19:13:59 +01:00
20e4a35814
[YouTube] Support richGridRenderer on channel pages
...
YouTube is deploying a new layout on their channel pages, which uses richGridRenderer JSON objects.
2022-11-02 19:01:29 +01:00
4cae66f1f9
Merge pull request #946 from chowder/dev
...
Add ability to identify short-form `StreamInfoItem`s
2022-11-01 12:19:58 +01:00
eb40bb8458
Merge pull request #959 from FireMasterK/playlist-info-item-uploader
...
Add uploaderUrl and uploaderVerified to PlaylistInfoItem.
2022-10-31 13:10:33 +01:00
b1a899fd47
Fix null pointer exception
2022-10-31 11:12:23 +00:00
a4db106a66
Merge pull request #960 from AudricV/yt-workaround-403-errors-android-client
...
[YouTube] Workaround 403 HTTP errors of ANDROID client streams
2022-10-30 21:53:26 +01:00
b441910257
Mark uploaderUrl as nullable.
2022-10-30 13:36:04 +00:00
3fdc0e72cc
Line breaks for long docstrings
2022-10-30 13:28:39 +00:00
6a256d0631
Add uploader url and verified to PlaylistInfoItem.
2022-10-30 13:00:19 +00:00
430504b4b5
Merge pull request #958 from AudricV/yt-playlists-support-new-metadata-format
...
[YouTube] Support new metadata format of playlists
2022-10-30 12:31:43 +01:00
f9bd08c649
Address reviews.
2022-10-30 01:25:30 +00:00
9282c3c13b
Fix exception message for YoutubeStreamInfoItemExtractor#isShortFormContent
...
Co-authored-by: AudricV <74829229+AudricV@users.noreply.github.com>
2022-10-30 01:23:15 +00:00
c5216f7c12
Update docstring for StreamExtractor#isShortFormContent
...
Co-authored-by: AudricV <74829229+AudricV@users.noreply.github.com>
2022-10-30 01:23:15 +00:00
04795fe5d2
Use Stream API for ShortFormContent#testShortFormContent
...
Co-authored-by: AudricV <74829229+AudricV@users.noreply.github.com>
2022-10-30 01:23:15 +00:00
975b38a0b9
Regenerate test mocks
2022-10-30 01:23:15 +00:00
4cccd33f3d
Implement isShortFormContent for StreamExtractor and StreamInfo
2022-10-30 01:23:15 +00:00
09544ceb23
Fix tests
2022-10-30 01:23:15 +00:00
644cc6cd76
Rename test function name
2022-10-30 01:23:15 +00:00
daf5674951
Add ability to identify short-form StreamInfoItems
2022-10-30 01:23:12 +00:00
3d314169b9
Merge pull request #943 from TeamNewPipe/fix/sc/comments
...
[SoundCloud] Fix getting more comments
2022-10-29 22:19:50 +02:00
65125a3eb4
[YouTube] Fix YoutubePlaylistExtractorTest.LearningPlaylist
...
The video count is now returned for this playlist, so it isn't unknown.
The testStreamCount method of this test class asserts now that the stream count
is greater than 40.
2022-10-29 18:12:10 +02:00
7258a53225
[YouTube] Support new playlist layout
...
This new layout doesn't provide author thumbnails and is completely different
for metadata, so the code to get them has been refactored.
The code of learning playlists video count check has been also removed, as it
seems to be not relevant anymore (the video count seems to be returned for
these playlists with both layouts).
Finally, unneeded overrides of subchannel methods, which don't apply to the
YouTube service, have been removed.
2022-10-29 18:12:10 +02:00
e37e8b358e
[YouTube] Update mocks that need to be updated
2022-10-29 18:10:18 +02:00
60e97cd274
[YouTube] Workaround getting streaming URLs returning 403 HTTP response codes
...
Using the player parameters used to get stories seems to fix the issue, which
affects currently only certain countries such as UK.
This is a workaround and should be fixed in a better way (by changing the
InnerTube additional client used for videos or finding what is now required in
Android player requests).
2022-10-29 17:58:33 +02:00
c230d84df1
Fix Checkstyle error in YoutubeCommentsInfoItemExtractor
2022-10-29 13:24:19 +02:00
0ffcb32d9c
[YouTube] Add comment reply count support ( #936 )
...
Add comment reply count support for YouTube and introduce `CommentsInfoItem.UNKNOWN_REPLY_COUNT` constant
Co-authored-by: AudricV <74829229+AudricV@users.noreply.github.com>
Co-authored-by: Tobi <TobiGr@users.noreply.github.com>
2022-10-15 12:40:06 +02:00
b90a566dd8
Add backport implementation of Locale.forLanguageTag().
2022-10-12 09:21:39 +05:30
b232c29d22
Use Locale.forLanguageTag().
2022-10-12 09:21:38 +05:30
4d136599bd
[SoundCloud] Fix getting more comments
2022-10-11 15:44:54 +02:00
a822e91909
Merge pull request #939 from TurtleArmyMc/fix/SoundcloudPlaylistExtractor_track_order
...
Fix SoundcloudPlaylistExtractor: tracks are in correct order
2022-10-10 22:28:18 +02:00
02810a7db7
Add a comment
2022-10-10 22:22:12 +02:00
54092fc3c7
Merge pull request #930 from Isira-Seneviratne/Avoid_Comparator_NPE
...
Avoid possible NullPointerException in MediaCCCRecentKiosk.
2022-10-10 11:43:34 +02:00
bf70d32eb4
Fix SoundcloudPlaylistExtractor: tracks are in correct order
2022-09-30 15:26:25 -04:00
8067c43837
[YouTube] Don't use a specific letter for the decryption function name pattern
...
Use the same possible characters for variables everywhere, in order to avoid
potential future throttling parameter decryption function name parsing issues
related to the usage of other letter(s) than b.
2022-09-24 21:49:22 +02:00
abcee87167
[YouTube] Fix throttling parameter decryption function regex
...
- Quote the function name, as it may contain special regex symbols, such as
dollar;
- Support multiple lines;
- Use what looks like the end of the function for the end of the regex (this
part is inspired from yt-dlp throttling parameter decryption regex);
- Move the throttling function body regex into a private and static constant.
2022-09-24 21:28:09 +02:00
7244be7627
[YouTube] Add JavaScript lexer to parse completely throttling decryption function ( #905 )
2022-09-24 19:55:46 +02:00
0e31c86aee
Avoid possible NullPointerException in MediaCCCRecentKiosk.
2022-09-21 05:40:07 +05:30
a99af9bb6e
Merge pull request #927 from Stypox/fix-feed-thumbnail
...
[YouTube] Return mqdefault thumbnails in fast feed
2022-09-19 08:52:01 +02:00
4e14644c41
Ensure comments are parsed with URLs and timestamps.
2022-09-17 16:03:39 +05:30
93ef73e2f2
[YouTube] Return mqdefault thumbnails in feed
...
The hqdefault thumbnails, which are used by default, have black bars at the top and at the bottom, and are thus not optimal.
2022-09-13 16:46:54 +02:00
d8ddeb549c
[YouTube] Support new trending structure and filter recently trending section
...
This new structure allow us to filter easily Trending shorts and Recently
trending sections.
On the previous one, this Recently trending section is now filtered, by
checking whether sections have a title, which isn't the case for normal trends
contrary to the other ones.
This makes that the extractor returns now only the real 50 "Now" YouTube
trends.
Elements inside arrays are now extracted dynamically instead of only the ones
of the first index, using Java 8's Stream API.
The getInitialPage() method of YoutubeTrendingExtractor can now throw a
ParsingException if no selected tab (corresponding to the one of the trends
type extracted) has been found.
Finally, the licence header has been moved to the top of the file and updated.
2022-09-11 18:52:12 +02:00
884da40f65
Merge pull request #926 from AudricV/fix-yt-like-count-extraction
...
[YouTube] Fix extraction of like count with the new data model
2022-09-11 12:21:38 +02:00
119b9129e2
[YouTube] Fix extraction of like count with the new data model
...
In the new data model currently A/B tested or deployed, the like count button
data is the same, but its path has been changed.
The extraction of the like count has been also improved, by using the multiple
accessibility data available instead of the only one which was used before.
This accessibility data has been also deprioritized, because it is language
dependent when there is no view.
Like count is always known, even for age-restricted videos, so returning -1 if
the like count extraction fails on this type of videos is not needed and hides
an extraction error: that's the reason why this return has been removed and the
exception will always be thrown, even if a video is age-restricted.
2022-09-10 23:28:38 +02:00
4905f74700
[YouTube] Fix throttling parameter decryption on Android
...
Escape the curly brace in the regular expression used to parse the throttling
parameter decryption function to allow its compatibility on Android.
2022-09-10 17:03:39 +02:00
fa25a7280f
Downgrade Rhino since it uses a class not available on Android
...
Reverts #774 , see TeamNewPipe/NewPipe#8876
2022-08-25 09:49:13 +02:00
25f0ce7305
Merge pull request #911 from TeamNewPipe/dependabot/gradle/org.jsoup-jsoup-1.15.3
...
Bump jsoup from 1.15.2 to 1.15.3
2022-08-24 14:23:44 +02:00
1d7a3a90a4
Bump jsoup from 1.15.2 to 1.15.3
...
Bumps [jsoup](https://github.com/jhy/jsoup ) from 1.15.2 to 1.15.3.
- [Release notes](https://github.com/jhy/jsoup/releases )
- [Changelog](https://github.com/jhy/jsoup/blob/master/CHANGES )
- [Commits](https://github.com/jhy/jsoup/compare/jsoup-1.15.2...jsoup-1.15.3 )
---
updated-dependencies:
- dependency-name: org.jsoup:jsoup
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2022-08-24 09:14:26 +00:00
41254ae12a
Remove annotation.
2022-08-24 06:59:17 +05:30
943b7c033b
Remove EMPTY_STRING.
2022-08-24 06:59:17 +05:30
d6577e5e0b
Merge pull request #882 from litetex/fix-all-tests
...
Fix all tests
2022-08-23 16:19:27 +02:00
03d9a4fe9d
Fix typos in YoutubeStreamExtractor.tryDecryptUrl
2022-08-21 21:17:03 +02:00
0beb55a232
Improve ``YoutubeThrottlingDecrypterTest``
2022-08-21 20:17:05 +02:00
9e93d6b193
YoutubeThrottlingDecrypter: Check if returned string is a valid JavaScript function
2022-08-21 20:16:45 +02:00
ed0a07af4e
YoutubeThrottlingDecrypter: Patch regex
2022-08-21 20:15:53 +02:00
641a447d9b
Updated consent-related mock-data
2022-08-21 18:43:48 +02:00
8ff7a90f52
Improved consent cookie related constants and documentation
2022-08-21 18:41:40 +02:00
cb64a480cd
[YouTube] Remove deprecated code in YoutubeThrottlingDecrypter
...
YoutubeThrottlingDecrypter is now a non-instantiable final class and non-static
attributes have been removed.
The static attributes of this class have been renamed, in order to respect the
naming specification used.
2022-08-16 15:01:05 +02:00
5bbea3a8f2
Fix ``YoutubeMixPlaylistExtractorTest`` again
...
Doesn't work for kurgesagt videos anymore, used music video (copyright free) instead
2022-08-14 15:12:27 +02:00
da06166065
Updated mock data of ``YoutubeSearchExtractorTest$Suggestion``
2022-08-14 15:12:26 +02:00
2e36ab1578
Fixed ``YoutubeSearchExtractorTest$Suggestion``
2022-08-14 15:12:26 +02:00
2a8a623643
``YoutubePlaylistLinkHandlerFactoryTest``: Use parameterized tests
2022-08-14 14:48:31 +02:00
938e69a16a
Fixed ``YoutubePlaylistLinkHandlerFactoryTest``
2022-08-14 14:48:30 +02:00
844de3e378
Fix ``YoutubeStreamLinkHandlerFactoryTest `` and parameterized the tests
...
used code from #833
2022-08-14 14:48:30 +02:00
c1a72b8240
Use Android Studios code style
2022-08-14 14:48:29 +02:00
fc27b8a5b8
Use preexisting ``ContentNotAvailableException``
2022-08-14 14:48:29 +02:00
ThetaDev
Kavin
dependabot[bot]
AudricV
Stypox
petlyh
fynngodau
Björn Sigurbergsson
TobiGr
Kunal
Isira Seneviratne
Luigi Russo
chowder
xz-dev
TurtleArmyMc
litetex