You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

269 lines
8.2 KiB

Revamp post filtering system (#18058) * Add model for custom filter keywords * Use CustomFilterKeyword internally Does not change the API * Fix /filters/edit and /filters/new * Add migration tests * Remove whole_word column from custom_filters (covered by custom_filter_keywords) * Redesign /filters Instead of a list, present a card that displays more information and handles multiple keywords per filter. * Redesign /filters/new and /filters/edit to add and remove keywords This adds a new gem dependency: cocoon, as well as a npm dependency: cocoon-js-vanilla. Those are used to easily populate and remove form fields from the user interface when manipulating multiple keyword filters at once. * Add /api/v2/filters to edit filter with multiple keywords Entities: - `Filter`: `id`, `title`, `filter_action` (either `hide` or `warn`), `context` `keywords` - `FilterKeyword`: `id`, `keyword`, `whole_word` API endpoits: - `GET /api/v2/filters` to list filters (including keywords) - `POST /api/v2/filters` to create a new filter `keywords_attributes` can also be passed to create keywords in one request - `GET /api/v2/filters/:id` to read a particular filter - `PUT /api/v2/filters/:id` to update a new filter `keywords_attributes` can also be passed to edit, delete or add keywords in one request - `DELETE /api/v2/filters/:id` to delete a particular filter - `GET /api/v2/filters/:id/keywords` to list keywords for a filter - `POST /api/v2/filters/:filter_id/keywords/:id` to add a new keyword to a filter - `GET /api/v2/filter_keywords/:id` to read a particular keyword - `PUT /api/v2/filter_keywords/:id` to edit a particular keyword - `DELETE /api/v2/filter_keywords/:id` to delete a particular keyword * Change from `irreversible` boolean to `action` enum * Remove irrelevent `irreversible_must_be_within_context` check * Fix /filters/new and /filters/edit with update for filter_action * Fix Rubocop/Codeclimate complaining about task names * Refactor FeedManager#phrase_filtered? This moves regexp building and filter caching to the `CustomFilter` class. This does not change the functional behavior yet, but this changes how the cache is built, doing per-custom_filter regexps so that filters can be matched independently, while still offering caching. * Perform server-side filtering and output result in REST API * Fix numerous filters_changed events being sent when editing multiple keywords at once * Add some tests * Use the new API in the WebUI - use client-side logic for filters we have fetched rules for. This is so that filter changes can be retroactively applied without reloading the UI. - use server-side logic for filters we haven't fetched rules for yet (e.g. network error, or initial timeline loading) * Minor optimizations and refactoring * Perform server-side filtering on the streaming server * Change the wording of filter action labels * Fix issues pointed out by linter * Change design of “Show anyway” link in accordence to review comments * Drop “irreversible” filtering behavior * Move /api/v2/filter_keywords to /api/v1/filters/keywords * Rename `filter_results` attribute to `filtered` * Rename REST::LegacyFilterSerializer to REST::V1::FilterSerializer * Fix systemChannelId value in streaming server * Simplify code by removing client-side filtering code The simplifcation comes at a cost though: filters aren't retroactively applied anymore.
1 year ago
Revamp post filtering system (#18058) * Add model for custom filter keywords * Use CustomFilterKeyword internally Does not change the API * Fix /filters/edit and /filters/new * Add migration tests * Remove whole_word column from custom_filters (covered by custom_filter_keywords) * Redesign /filters Instead of a list, present a card that displays more information and handles multiple keywords per filter. * Redesign /filters/new and /filters/edit to add and remove keywords This adds a new gem dependency: cocoon, as well as a npm dependency: cocoon-js-vanilla. Those are used to easily populate and remove form fields from the user interface when manipulating multiple keyword filters at once. * Add /api/v2/filters to edit filter with multiple keywords Entities: - `Filter`: `id`, `title`, `filter_action` (either `hide` or `warn`), `context` `keywords` - `FilterKeyword`: `id`, `keyword`, `whole_word` API endpoits: - `GET /api/v2/filters` to list filters (including keywords) - `POST /api/v2/filters` to create a new filter `keywords_attributes` can also be passed to create keywords in one request - `GET /api/v2/filters/:id` to read a particular filter - `PUT /api/v2/filters/:id` to update a new filter `keywords_attributes` can also be passed to edit, delete or add keywords in one request - `DELETE /api/v2/filters/:id` to delete a particular filter - `GET /api/v2/filters/:id/keywords` to list keywords for a filter - `POST /api/v2/filters/:filter_id/keywords/:id` to add a new keyword to a filter - `GET /api/v2/filter_keywords/:id` to read a particular keyword - `PUT /api/v2/filter_keywords/:id` to edit a particular keyword - `DELETE /api/v2/filter_keywords/:id` to delete a particular keyword * Change from `irreversible` boolean to `action` enum * Remove irrelevent `irreversible_must_be_within_context` check * Fix /filters/new and /filters/edit with update for filter_action * Fix Rubocop/Codeclimate complaining about task names * Refactor FeedManager#phrase_filtered? This moves regexp building and filter caching to the `CustomFilter` class. This does not change the functional behavior yet, but this changes how the cache is built, doing per-custom_filter regexps so that filters can be matched independently, while still offering caching. * Perform server-side filtering and output result in REST API * Fix numerous filters_changed events being sent when editing multiple keywords at once * Add some tests * Use the new API in the WebUI - use client-side logic for filters we have fetched rules for. This is so that filter changes can be retroactively applied without reloading the UI. - use server-side logic for filters we haven't fetched rules for yet (e.g. network error, or initial timeline loading) * Minor optimizations and refactoring * Perform server-side filtering on the streaming server * Change the wording of filter action labels * Fix issues pointed out by linter * Change design of “Show anyway” link in accordence to review comments * Drop “irreversible” filtering behavior * Move /api/v2/filter_keywords to /api/v1/filters/keywords * Rename `filter_results` attribute to `filtered` * Rename REST::LegacyFilterSerializer to REST::V1::FilterSerializer * Fix systemChannelId value in streaming server * Simplify code by removing client-side filtering code The simplifcation comes at a cost though: filters aren't retroactively applied anymore.
1 year ago
Spelling (#17705) * spelling: account Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> * spelling: affiliated Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> * spelling: appearance Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> * spelling: autosuggest Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> * spelling: cacheable Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> * spelling: component Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> * spelling: conversations Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> * spelling: domain.example Clarify what's distinct and use RFC friendly domain space. Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> * spelling: environment Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> * spelling: exceeds Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> * spelling: functional Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> * spelling: inefficiency Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> * spelling: not Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> * spelling: notifications Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> * spelling: occurring Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> * spelling: position Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> * spelling: progress Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> * spelling: promotable Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> * spelling: reblogging Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> * spelling: repetitive Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> * spelling: resolve Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> * spelling: saturated Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> * spelling: similar Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> * spelling: strategies Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> * spelling: success Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> * spelling: targeting Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> * spelling: thumbnails Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> * spelling: unauthorized Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> * spelling: unsensitizes Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> * spelling: validations Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> * spelling: various Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> Co-authored-by: Josh Soref <jsoref@users.noreply.github.com>
2 years ago
Spelling (#17705) * spelling: account Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> * spelling: affiliated Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> * spelling: appearance Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> * spelling: autosuggest Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> * spelling: cacheable Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> * spelling: component Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> * spelling: conversations Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> * spelling: domain.example Clarify what's distinct and use RFC friendly domain space. Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> * spelling: environment Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> * spelling: exceeds Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> * spelling: functional Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> * spelling: inefficiency Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> * spelling: not Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> * spelling: notifications Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> * spelling: occurring Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> * spelling: position Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> * spelling: progress Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> * spelling: promotable Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> * spelling: reblogging Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> * spelling: repetitive Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> * spelling: resolve Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> * spelling: saturated Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> * spelling: similar Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> * spelling: strategies Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> * spelling: success Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> * spelling: targeting Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> * spelling: thumbnails Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> * spelling: unauthorized Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> * spelling: unsensitizes Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> * spelling: validations Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> * spelling: various Signed-off-by: Josh Soref <jsoref@users.noreply.github.com> Co-authored-by: Josh Soref <jsoref@users.noreply.github.com>
2 years ago
  1. require 'rails_helper'
  2. RSpec.describe Api::V1::StatusesController, type: :controller do
  3. render_views
  4. let(:user) { Fabricate(:user) }
  5. let(:app) { Fabricate(:application, name: 'Test app', website: 'http://testapp.com') }
  6. let(:token) { Fabricate(:accessible_access_token, resource_owner_id: user.id, application: app, scopes: scopes) }
  7. context 'with an oauth token' do
  8. before do
  9. allow(controller).to receive(:doorkeeper_token) { token }
  10. end
  11. describe 'GET #show' do
  12. let(:scopes) { 'read:statuses' }
  13. let(:status) { Fabricate(:status, account: user.account) }
  14. it 'returns http success' do
  15. get :show, params: { id: status.id }
  16. expect(response).to have_http_status(200)
  17. end
  18. context 'when post includes filtered terms' do
  19. let(:status) { Fabricate(:status, text: 'this toot is about that banned word') }
  20. before do
  21. user.account.custom_filters.create!(phrase: 'filter1', context: %w(home), action: :hide, keywords_attributes: [{ keyword: 'banned' }, { keyword: 'irrelevant' }])
  22. end
  23. it 'returns http success' do
  24. get :show, params: { id: status.id }
  25. expect(response).to have_http_status(200)
  26. end
  27. it 'returns filter information' do
  28. get :show, params: { id: status.id }
  29. json = body_as_json
  30. expect(json[:filtered][0]).to include({
  31. filter: a_hash_including({
  32. id: user.account.custom_filters.first.id.to_s,
  33. title: 'filter1',
  34. filter_action: 'hide',
  35. }),
  36. keyword_matches: ['banned'],
  37. })
  38. end
  39. end
  40. context 'when post is explicitly filtered' do
  41. let(:status) { Fabricate(:status, text: 'hello world') }
  42. before do
  43. filter = user.account.custom_filters.create!(phrase: 'filter1', context: %w(home), action: :hide)
  44. filter.statuses.create!(status_id: status.id)
  45. end
  46. it 'returns http success' do
  47. get :show, params: { id: status.id }
  48. expect(response).to have_http_status(200)
  49. end
  50. it 'returns filter information' do
  51. get :show, params: { id: status.id }
  52. json = body_as_json
  53. expect(json[:filtered][0]).to include({
  54. filter: a_hash_including({
  55. id: user.account.custom_filters.first.id.to_s,
  56. title: 'filter1',
  57. filter_action: 'hide',
  58. }),
  59. status_matches: [status.id.to_s],
  60. })
  61. end
  62. end
  63. context 'when reblog includes filtered terms' do
  64. let(:status) { Fabricate(:status, reblog: Fabricate(:status, text: 'this toot is about that banned word')) }
  65. before do
  66. user.account.custom_filters.create!(phrase: 'filter1', context: %w(home), action: :hide, keywords_attributes: [{ keyword: 'banned' }, { keyword: 'irrelevant' }])
  67. end
  68. it 'returns http success' do
  69. get :show, params: { id: status.id }
  70. expect(response).to have_http_status(200)
  71. end
  72. it 'returns filter information' do
  73. get :show, params: { id: status.id }
  74. json = body_as_json
  75. expect(json[:reblog][:filtered][0]).to include({
  76. filter: a_hash_including({
  77. id: user.account.custom_filters.first.id.to_s,
  78. title: 'filter1',
  79. filter_action: 'hide',
  80. }),
  81. keyword_matches: ['banned'],
  82. })
  83. end
  84. end
  85. end
  86. describe 'GET #context' do
  87. let(:scopes) { 'read:statuses' }
  88. let(:status) { Fabricate(:status, account: user.account) }
  89. before do
  90. Fabricate(:status, account: user.account, thread: status)
  91. end
  92. it 'returns http success' do
  93. get :context, params: { id: status.id }
  94. expect(response).to have_http_status(200)
  95. end
  96. end
  97. describe 'POST #create' do
  98. let(:scopes) { 'write:statuses' }
  99. context do
  100. before do
  101. post :create, params: { status: 'Hello world' }
  102. end
  103. it 'returns http success' do
  104. expect(response).to have_http_status(200)
  105. end
  106. it 'returns rate limit headers' do
  107. expect(response.headers['X-RateLimit-Limit']).to eq RateLimiter::FAMILIES[:statuses][:limit].to_s
  108. expect(response.headers['X-RateLimit-Remaining']).to eq (RateLimiter::FAMILIES[:statuses][:limit] - 1).to_s
  109. end
  110. end
  111. context 'with a safeguard' do
  112. let!(:alice) { Fabricate(:account, username: 'alice') }
  113. let!(:bob) { Fabricate(:account, username: 'bob') }
  114. before do
  115. post :create, params: { status: '@alice hm, @bob is really annoying lately', allowed_mentions: [alice.id] }
  116. end
  117. it 'returns http unprocessable entity' do
  118. expect(response).to have_http_status(422)
  119. end
  120. it 'returns serialized extra accounts in body' do
  121. expect(body_as_json[:unexpected_accounts].map { |a| a.slice(:id, :acct) }).to eq [{ id: bob.id.to_s, acct: bob.acct }]
  122. end
  123. end
  124. context 'with missing parameters' do
  125. before do
  126. post :create, params: {}
  127. end
  128. it 'returns http unprocessable entity' do
  129. expect(response).to have_http_status(422)
  130. end
  131. it 'returns rate limit headers' do
  132. expect(response.headers['X-RateLimit-Limit']).to eq RateLimiter::FAMILIES[:statuses][:limit].to_s
  133. end
  134. end
  135. context 'when exceeding rate limit' do
  136. before do
  137. rate_limiter = RateLimiter.new(user.account, family: :statuses)
  138. 300.times { rate_limiter.record! }
  139. post :create, params: { status: 'Hello world' }
  140. end
  141. it 'returns http too many requests' do
  142. expect(response).to have_http_status(429)
  143. end
  144. it 'returns rate limit headers' do
  145. expect(response.headers['X-RateLimit-Limit']).to eq RateLimiter::FAMILIES[:statuses][:limit].to_s
  146. expect(response.headers['X-RateLimit-Remaining']).to eq '0'
  147. end
  148. end
  149. end
  150. describe 'DELETE #destroy' do
  151. let(:scopes) { 'write:statuses' }
  152. let(:status) { Fabricate(:status, account: user.account) }
  153. before do
  154. post :destroy, params: { id: status.id }
  155. end
  156. it 'returns http success' do
  157. expect(response).to have_http_status(200)
  158. end
  159. it 'removes the status' do
  160. expect(Status.find_by(id: status.id)).to be nil
  161. end
  162. end
  163. describe 'PUT #update' do
  164. let(:scopes) { 'write:statuses' }
  165. let(:status) { Fabricate(:status, account: user.account) }
  166. before do
  167. put :update, params: { id: status.id, status: 'I am updated' }
  168. end
  169. it 'returns http success' do
  170. expect(response).to have_http_status(200)
  171. end
  172. it 'updates the status' do
  173. expect(status.reload.text).to eq 'I am updated'
  174. end
  175. end
  176. end
  177. context 'without an oauth token' do
  178. before do
  179. allow(controller).to receive(:doorkeeper_token) { nil }
  180. end
  181. context 'with a private status' do
  182. let(:status) { Fabricate(:status, account: user.account, visibility: :private) }
  183. describe 'GET #show' do
  184. it 'returns http unauthorized' do
  185. get :show, params: { id: status.id }
  186. expect(response).to have_http_status(404)
  187. end
  188. end
  189. describe 'GET #context' do
  190. before do
  191. Fabricate(:status, account: user.account, thread: status)
  192. end
  193. it 'returns http unauthorized' do
  194. get :context, params: { id: status.id }
  195. expect(response).to have_http_status(404)
  196. end
  197. end
  198. end
  199. context 'with a public status' do
  200. let(:status) { Fabricate(:status, account: user.account, visibility: :public) }
  201. describe 'GET #show' do
  202. it 'returns http success' do
  203. get :show, params: { id: status.id }
  204. expect(response).to have_http_status(200)
  205. end
  206. end
  207. describe 'GET #context' do
  208. before do
  209. Fabricate(:status, account: user.account, thread: status)
  210. end
  211. it 'returns http success' do
  212. get :context, params: { id: status.id }
  213. expect(response).to have_http_status(200)
  214. end
  215. end
  216. end
  217. end
  218. end