Commit Graph

1961 Commits

Author SHA1 Message Date
TobiGr 7c29dbc965 Fix JDoc
Add Checkstyle exception for LineLength in JDoc links.

See https://github.com/TeamNewPipe/NewPipeExtractor/actions/runs/8804403691/job/24164703883
2024-04-23 19:55:51 +02:00
Stypox 4e9e7cb29c
Improve assertTabsContain() to also check size 2024-04-20 11:48:36 +02:00
Stypox 9d0dd36034
[YouTube] Create constants for client names/versions 2024-04-20 11:43:54 +02:00
Stypox d4e6d22e64
[YouTube] Improve meta info code for review 2024-04-20 11:43:08 +02:00
AudricV 74bf000473
[YouTube] Update mocks 2024-04-11 18:59:11 +02:00
AudricV f9792cf3a9
[YouTube] Fix InteractiveTabbedHeader.testTabs test
YouTube now returns a Shorts tab for InteractiveTabbedHeader gaming channels,
which contains Shorts about the game of the topic channel but are not uploaded
on the game topic channel.

As this tab is already supported by the extractor, fetching a gaming topic
channel now returns a tab instead of none.

Channel name and channel URL of these Shorts needs to be set to null in a
separate commit, as Shorts on this tab do not have the topic channel as their
uploader.
2024-04-11 18:59:10 +02:00
AudricV f40fc0aa4f
[YouTube] Add support for new crisis meta info action data
The new action data can return multiple contact actions instead of only one,
which will be concatenated by a new line return.

This commit fixes tests of the CrisisResources test class of
YoutubeSearchExtractorTest.
2024-04-11 18:59:09 +02:00
AudricV 2a3c6f80d2
[YouTube] Fix YoutubeStreamExtractorRelatedMixTest 2024-04-10 21:19:03 +02:00
AudricV 657b4377aa
[YouTube] Fix YoutubeStreamExtractorDefaultTest tests 2024-04-10 21:19:03 +02:00
AudricV 7bf50bf1cb
[YouTube] Update Android client player parameters
YouTube disabled the effectiveness of the parameters which were used (the
player response we get redirects to another video), but new parameters which
work around Android's client integrity checks have been found.
2024-04-10 21:19:03 +02:00
AudricV 27dc1b1f50
[YouTube] Remove usage of API keys for InnerTube requests, bump versions
The API keys are not used anymore by official clients in almost all cases
(still used by the Android app until it gets a configuration) for all requests
we made.

Clients and device OS versions have been bumped to their latest stable version
known.

Methods and fields related to API keys have been renamed or deleted if they're
no longer relevant.
2024-04-10 21:19:02 +02:00
AudricV e380bb4bc3
[YouTube] Add missing prettyPrint query parameter to mixes continuations 2024-04-10 19:06:36 +02:00
Stypox 02274d5395
[YouTube] Avoid XSS attacks in description or comments 2024-04-08 11:21:31 +02:00
Stypox 3f7b2653e3
[YouTube] Add YoutubeDescriptionHelperTest 2024-04-08 11:21:31 +02:00
Stypox a90237816a
[YouTube] Cleanup description helper
Remove unneeded isClose field, and make constants private
2024-04-08 11:21:31 +02:00
Stypox b80c3f5d51
[YouTube] Replace link text with accessibility label 2024-04-08 00:14:28 +02:00
Stypox 09732d6785
[YouTube] Add support for styles in attributed descriptions
Also refactor descriptions parsing.
2024-04-04 21:14:27 +02:00
AudricV 293c3e9e47
[YouTube] Support new A/B tested comments data
Also improve current comments code by removing outdated comment
renderer data.
2024-04-04 21:14:26 +02:00
Stypox e5b30ae8c3
Merge pull request #1151 from Profpatsch/localization-return-optional
LocaleCompat.forLanguageTag: return Optional if parsing fails
2024-03-29 13:50:48 +01:00
Stypox 23fc7aa209
Throw ParsingException instead of IllegalArg 2024-03-29 13:44:42 +01:00
Stypox fb468a23f4
Merge pull request #1142 from TeamNewPipe/peertube-v6
[PeerTube] Add support for PeerTube v6 features
2024-03-29 12:25:38 +01:00
Stypox 6589e2c15d
Merge pull request #1148 from Stypox/mediaccc-channel-tab-handler
[MediaCCC] Allow obtaining channel tab link handler
2024-03-28 13:45:05 +01:00
Stypox c57016b79b
Make getCommentText @Nonnull 2024-03-27 15:26:06 +01:00
TobiGr 51ddacc81d [SoundCloud] Fix SoundcloudSearchExtractorTest.NoNextPage
Search did not return no item at all, causing a NothingFoundException. New search query yields three items on first page
2024-03-20 15:10:39 +01:00
TobiGr 8392d50ba6 Update mocks for YoutubeChannelExtractorTest.NotAvailable 2024-03-20 14:59:44 +01:00
TobiGr aaccfecda8 [YouTube] Detect new account termination messages 2024-03-20 14:57:41 +01:00
TobiGr 73f0c63a9d [PeerTube] Fix tests for "What is PeerTube?" video 2024-03-20 14:44:06 +01:00
TobiGr fe47a4311f [PeerTube] Add test for segments and framesets 2024-01-29 10:22:06 +01:00
TobiGr 15e0e74b48 [PeerTube] Add support for stream frames/storyboards extraction
Implement PeerTubeStreamExtractor.getFrames()
2024-01-29 10:22:06 +01:00
Profpatsch 7408173246 LocaleCompat.forLanguageTag: return Optional if parsing fails
It’s not obvious that the function will fail in some cases and throw
an `IllegalArgumentException`.

So instead of just failing if parsing fails, return an Optional that
all callers have to decide what to do (e.g. the YoutubeExtractor can
just ignore the locale in that case, like it does with most other
fields in the json if they are unexpected).
2024-01-07 14:31:34 +01:00
Stypox aaf3231fc7
[MediaCCC] Fix lambda link handler keeping reference to extractor
This caused problems in NewPipe, because extractors are not serializable, and well, keeping references to them is a bad idea anyway.
2023-12-30 23:23:19 +01:00
Stypox 137e924035
[MediaCCC] Add ChannelTabExtractorTest 2023-12-30 22:53:51 +01:00
Stypox cc9ade962e
[MediaCCC] Allow obtaining channel tab extractor from scratch
i.e. without needing to pass through the conference/channel extractor
This was needed because clients (like NewPipe) might rely on link handlers to hold as little data as possible, since they might be kept around for long or passed around in system transactions, so this commit allows obtaining a standalone link handler that does not hold a JsonObject within itself.
2023-12-30 22:53:27 +01:00
petlyh 6dc25f7b97
[YouTube] Add Albums channel tab mocks 2023-12-30 14:46:39 +01:00
petlyh 4408e2d0ac
[YouTube] Add Albums channel tab 2023-12-30 14:01:30 +01:00
TobiGr 9ab932e394 Rename testDoNotAcceptNonURLs() -> assertDoNotAcceptNonURLs() 2023-12-29 16:38:11 +01:00
TobiGr 61d237de02 [PeerTube] Test onAccept(String URL) in LinkHandlerFactories for non-URLs 2023-12-29 12:45:02 +01:00
petlyh 2b2c1546d1 Avoid PeerTube accepting non-URLs 2023-12-29 12:27:39 +01:00
Tobi 1e93b1dc20
Merge pull request #1135 from Stypox/yt-emergency-info
[YouTube] Implement emergency meta info
2023-12-29 12:01:40 +01:00
TobiGr 8f9ebdcb77 [PeerTube] Fix failing PeertubeTrendingLinkHandlerFactoryTest
The factory was updated in #1144
2023-12-29 11:52:19 +01:00
Stypox 1553931027
Merge pull request #1145 from TeamNewPipe/dependabot/gradle/org.jsoup-jsoup-1.17.2
Bump org.jsoup:jsoup from 1.16.2 to 1.17.2
2023-12-29 11:27:01 +01:00
dependabot[bot] 151ee99da3
Bump org.jsoup:jsoup from 1.16.2 to 1.17.2
Bumps [org.jsoup:jsoup](https://github.com/jhy/jsoup) from 1.16.2 to 1.17.2.
- [Release notes](https://github.com/jhy/jsoup/releases)
- [Changelog](https://github.com/jhy/jsoup/blob/master/CHANGES.md)
- [Commits](https://github.com/jhy/jsoup/compare/jsoup-1.16.2...jsoup-1.17.2)

---
updated-dependencies:
- dependency-name: org.jsoup:jsoup
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-12-29 09:47:00 +00:00
dragfyre 65e7bc5b95
Update PeertubeTrendingLinkHandlerFactory.java
correcting Peertube local trending api URL (per #10685 in main NewPipe repo); see https://docs.joinpeertube.org/api-rest-reference.html#tag/Video/operation/getVideos
2023-12-28 14:50:31 +07:00
Stypox fc54fb2fdb
Merge pull request #1140 from Stypox/yt-shorts-no-duration
[YouTube] Always return -1 as duration of Shorts returned inside reel items
2023-12-21 21:52:40 +01:00
Stypox 0518487d26
Fix SearchInfo's non-null MetaInfo being null when initialized or when an extraction error occurs (#1141)
The meta info might have been null either when SearchInfo is first initialized, or when extractor.getMetaInfo() throws an exception in getInfo().
This caused NewPipe to crash instead of showing a nice error in https://www.reddit.com/r/youtube/comments/184ttmw/what_exactly_about_blue_whales_has_youtube_so/.
2023-12-21 21:39:25 +01:00
Stypox 5b59a1a8c5
[YouTube] Move meta info extraction to separate file
YoutubeParsingHelper was longer than 2000 lines which caused checkstyle issues
2023-12-21 21:19:08 +01:00
Stypox b8e12dd76c
[YouTube] Implement emergency meta info
YouTube provides that meta info panel when users search for really sensitive content like suicide (e.g. "blue whale").

It contains:
- an encouragement as title (e.g. "We are with you")
- a phone number as action
- details about how to call the phone number (e.g. availability)
- an url pointing to the website of an association

Also add a test that just checks if a meta info is properly extracted
2023-12-21 21:19:08 +01:00
Stypox 83c1737f70
[YouTube] Update shorts channel tab mocks 2023-12-21 20:54:16 +01:00
Stypox 2938067c2c
[YouTube] Shorts don't provide a duration anymore 2023-12-21 20:41:01 +01:00
AudricV ec0194cfbf
[YouTube] Update mocks 2023-12-08 21:46:52 +01:00