Browse Source

Support unicode emojis and remove emojify.js (#11032)

* Support unicode emojis and remove emojify.js

This PR replaces all use of emojify.js and adds unicode emoji support to various areas of gitea.

This works in a few ways:

First it adds emoji parsing support into gitea itself. This allows us to

 * Render emojis from valid alias (😄)
 * Detect unicode emojis and let us put them in their own class with proper aria-labels and styling
 * Easily allow for custom "emoji"
 * Support all emoji rendering and features without javascript
 * Uses plain unicode and lets the system render in appropriate emoji font
 * Doesn't leave us relying on external sources for updates/fixes/features

That same list of emoji is also used to create a json file which replaces the part of emojify.js that populates the emoji search tribute. This file is about 35KB with GZIP turned on and I've set it to load after the page renders to not hinder page load time (and this removes loading emojify.js also)

For custom "emoji" it uses a pretty simple scheme of just looking for /emojis/img/name.png where name is something a user has put in the "allowed reactions" setting we already have. The gitea reaction that was previously hard coded into a forked copy of emojify.js is included and works as a custom reaction under this method.

The emoji data sourced here is from https://github.com/github/gemoji which is the gem library Github uses for their emoji rendering (and a data source for other sites). So we should be able to easily render any emoji and :alias: that Github can, removing any errors from migrated content. They also update it as well, so we can sync when there are new unicode emoji lists released.

I've included a slimmed down and slightly modified forked copy of https://github.com/knq/emoji to make up our own emoji module. The code is pretty straight forward and again allows us to have a lot of flexibility in what happens.

I had seen a few comments about performance in some of the other threads if we render this ourselves, but there doesn't seem to be any issue here. In a test it can parse, convert, and render 1,000 emojis inside of a large markdown table in about 100ms on my laptop (which is many more emojis than will ever be in any normal issue). This also prevents any flickering and other weirdness from using javascript to render some things while using go for others.

Not included here are image fall back URLS. I don't really think they are necessary for anything new being written in 2020. However, managing the emoji ourselves would allow us to add these as a feature later on if it seems necessary.

Fixes: https://github.com/go-gitea/gitea/issues/9182
Fixes: https://github.com/go-gitea/gitea/issues/8974
Fixes: https://github.com/go-gitea/gitea/issues/8953
Fixes: https://github.com/go-gitea/gitea/issues/6628
Fixes: https://github.com/go-gitea/gitea/issues/5130

* add new shared function emojiHTML

* don't increase emoji size in issue title

* Update templates/repo/issue/view_content/add_reaction.tmpl

Co-Authored-By: 6543 <6543@obermui.de>

* Support for emoji rendering in various templates

* Render code and review comments as they should be

* Better way to handle mail subjects

* insert unicode from tribute selection

* Add template helper for plain text when needed

* Use existing replace function I forgot about

* Don't include emoji greater than Unicode Version 12

Only include emoji and aliases in JSON

* Update build/generate-emoji.go

* Tweak regex slightly to really match everything including random invisible characters. Run tests for every emoji we have

* final updates

* code review

* code review

* hard code gitea custom emoji to match previous behavior

* Update .eslintrc

Co-Authored-By: silverwind <me@silverwind.io>

* disable preempt

Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: Lauris BH <lauris@nix.lv>
Co-authored-by: guillep2k <18600385+guillep2k@users.noreply.github.com>
for-closed-social
mrsdizzie 4 years ago
committed by GitHub
parent
commit
4563eb873d
No known key found for this signature in database GPG Key ID: 4AEE18F83AFDEB23
928 changed files with 2713 additions and 167 deletions
  1. +1
    -1
      .eslintrc
  2. +1
    -0
      assets/emoji.json
  3. +184
    -0
      build/generate-emoji.go
  4. +3
    -2
      custom/conf/app.ini.sample
  5. +3
    -1
      docs/content/doc/advanced/config-cheat-sheet.en-us.md
  6. +0
    -1
      docs/content/page/index.en-us.md
  7. +0
    -1
      docs/content/page/index.fr-fr.md
  8. +0
    -1
      docs/content/page/index.zh-cn.md
  9. +0
    -1
      docs/content/page/index.zh-tw.md
  10. +119
    -0
      modules/emoji/emoji.go
  11. +1734
    -0
      modules/emoji/emoji_data.go
  12. +67
    -0
      modules/emoji/emoji_test.go
  13. +0
    -1
      modules/git/command.go
  14. +123
    -0
      modules/markup/html.go
  15. +45
    -0
      modules/markup/html_test.go
  16. +4
    -0
      modules/markup/sanitizer.go
  17. +27
    -0
      modules/templates/helper.go
  18. BIN
     
  19. +201
    -0
      public/vendor/assets/noto-color-emoji/LICENSE
  20. BIN
     
  21. +0
    -5
      public/vendor/librejs.html
  22. +0
    -8
      public/vendor/plugins/emojify/LICENSE
  23. +0
    -1
      public/vendor/plugins/emojify/emojify.custom.js
  24. BIN
     
  25. BIN
     
  26. BIN
     
  27. BIN
     
  28. BIN
     
  29. BIN
     
  30. BIN
     
  31. BIN
     
  32. BIN
     
  33. BIN
     
  34. BIN
     
  35. BIN
     
  36. BIN
     
  37. BIN
     
  38. BIN
     
  39. BIN
     
  40. BIN
     
  41. BIN
     
  42. BIN
     
  43. BIN
     
  44. BIN
     
  45. BIN
     
  46. BIN
     
  47. BIN
     
  48. BIN
     
  49. BIN
     
  50. BIN
     
  51. BIN
     
  52. BIN
     
  53. BIN
     
  54. BIN
     
  55. BIN
     
  56. BIN
     
  57. BIN
     
  58. BIN
     
  59. BIN
     
  60. BIN
     
  61. BIN
     
  62. BIN
     
  63. BIN
     
  64. BIN
     
  65. BIN
     
  66. BIN
     
  67. BIN
     
  68. BIN
     
  69. BIN
     
  70. BIN
     
  71. BIN
     
  72. BIN
     
  73. BIN
     
  74. BIN
     
  75. BIN
     
  76. BIN
     
  77. BIN
     
  78. BIN
     
  79. BIN
     
  80. BIN
     
  81. BIN
     
  82. BIN
     
  83. BIN
     
  84. BIN
     
  85. BIN
     
  86. BIN
     
  87. BIN
     
  88. BIN
     
  89. BIN
     
  90. BIN
     
  91. BIN
     
  92. BIN
     
  93. BIN
     
  94. BIN
     
  95. BIN
     
  96. BIN
     
  97. BIN
     
  98. BIN
     
  99. BIN
     
  100. BIN
     

+ 1
- 1
.eslintrc View File

@ -20,9 +20,9 @@ globals:
__webpack_public_path__: true
CodeMirror: false
Dropzone: false
emojify: false
SimpleMDE: false
u2fApi: false
Tribute: false
overrides:
- files: ["web_src/**/*.worker.js"]

+ 1
- 0
assets/emoji.json
File diff suppressed because it is too large
View File


+ 184
- 0
build/generate-emoji.go View File

@ -0,0 +1,184 @@
// Copyright 2020 The Gitea Authors. All rights reserved.
// Copyright 2015 Kenneth Shaw
// Use of this source code is governed by a MIT-style
// license that can be found in the LICENSE file.
// +build ignore
package main
import (
"encoding/json"
"flag"
"fmt"
"go/format"
"io/ioutil"
"log"
"net/http"
"regexp"
"sort"
"strconv"
"strings"
)
const (
gemojiURL = "https://raw.githubusercontent.com/github/gemoji/master/db/emoji.json"
maxUnicodeVersion = 12
)
var (
flagOut = flag.String("o", "modules/emoji/emoji_data.go", "out")
)
// Gemoji is a set of emoji data.
type Gemoji []Emoji
// Emoji represents a single emoji and associated data.
type Emoji struct {
Emoji string `json:"emoji"`
Description string `json:"description,omitempty"`
Aliases []string `json:"aliases"`
UnicodeVersion string `json:"unicode_version,omitempty"`
}
// Don't include some fields in JSON
func (e Emoji) MarshalJSON() ([]byte, error) {
type emoji Emoji
x := emoji(e)
x.UnicodeVersion = ""
x.Description = ""
return json.Marshal(x)
}
func main() {
var err error
flag.Parse()
// generate data
buf, err := generate()
if err != nil {
log.Fatal(err)
}
// write
err = ioutil.WriteFile(*flagOut, buf, 0644)
if err != nil {
log.Fatal(err)
}
}
var replacer = strings.NewReplacer(
"main.Gemoji", "Gemoji",
"main.Emoji", "\n",
"}}", "},\n}",
", Description:", ", ",
", Aliases:", ", ",
", UnicodeVersion:", ", ",
)
var emojiRE = regexp.MustCompile(`\{Emoji:"([^"]*)"`)
func generate() ([]byte, error) {
var err error
// load gemoji data
res, err := http.Get(gemojiURL)
if err != nil {
return nil, err
}
defer res.Body.Close()
// read all
body, err := ioutil.ReadAll(res.Body)
if err != nil {
return nil, err
}
// unmarshal
var data Gemoji
err = json.Unmarshal(body, &data)
if err != nil {
return nil, err
}
var re = regexp.MustCompile(`keycap|registered|copyright`)
tmp := data[:0]
// filter out emoji that require greater than max unicode version
for i := range data {
val, _ := strconv.ParseFloat(data[i].UnicodeVersion, 64)
if int(val) <= maxUnicodeVersion {
// remove these keycaps for now they really complicate matching since
// they include normal letters in them
if re.MatchString(data[i].Description) {
continue
}
tmp = append(tmp, data[i])
}
}
data = tmp
sort.Slice(data, func(i, j int) bool {
return data[i].Aliases[0] < data[j].Aliases[0]
})
aliasPairs := make([]string, 0)
aliasMap := make(map[string]int, len(data))
for i, e := range data {
if e.Emoji == "" || len(e.Aliases) == 0 {
continue
}
for _, a := range e.Aliases {
if a == "" {
continue
}
aliasMap[a] = i
aliasPairs = append(aliasPairs, ":"+a+":", e.Emoji)
}
}
// gitea customizations
i, ok := aliasMap["tada"]
if ok {
data[i].Aliases = append(data[i].Aliases, "hooray")
}
i, ok = aliasMap["laughing"]
if ok {
data[i].Aliases = append(data[i].Aliases, "laugh")
}
// add header
str := replacer.Replace(fmt.Sprintf(hdr, gemojiURL, data))
// change the format of the unicode string
str = emojiRE.ReplaceAllStringFunc(str, func(s string) string {
var err error
s, err = strconv.Unquote(s[len("{Emoji:"):])
if err != nil {
panic(err)
}
return "{" + strconv.QuoteToASCII(s)
})
// write a JSON file to use with tribute
file, _ := json.Marshal(data)
_ = ioutil.WriteFile("assets/emoji.json", file, 0644)
// format
return format.Source([]byte(str))
}
const hdr = `
// Copyright 2020 The Gitea Authors. All rights reserved.
// Use of this source code is governed by a MIT-style
// license that can be found in the LICENSE file.
package emoji
// Code generated by gen.go. DO NOT EDIT.
// Sourced from %s
//
var GemojiData = %#v
`

+ 3
- 2
custom/conf/app.ini.sample View File

@ -171,8 +171,9 @@ SHOW_USER_EMAIL = true
DEFAULT_THEME = gitea
; All available themes. Allow users select personalized themes regardless of the value of `DEFAULT_THEME`.
THEMES = gitea,arc-green
; All available reactions. Allow users react with different emoji's
; For the whole list look at https://gitea.com/gitea/gitea.com/issues/8
;All available reactions users can choose on issues/prs and comments.
;Values can be emoji alias (:smile:) or a unicode emoji.
;For custom reactions, add a tightly cropped square image to public/emoji/img/reaction_name.png
REACTIONS = +1, -1, laugh, hooray, confused, heart, rocket, eyes
; Whether the full name of the users should be shown where possible. If the full name isn't set, the username will be used.
DEFAULT_SHOW_FULL_NAME = false

+ 3
- 1
docs/content/doc/advanced/config-cheat-sheet.en-us.md View File

@ -128,7 +128,9 @@ Values containing `#` or `;` must be quoted using `` ` `` or `"""`.
- `DEFAULT_THEME`: **gitea**: \[gitea, arc-green\]: Set the default theme for the Gitea install.
- `THEMES`: **gitea,arc-green**: All available themes. Allow users select personalized themes
regardless of the value of `DEFAULT_THEME`.
- `REACTIONS`: All available reactions. Allow users react with different emoji's.
- `REACTIONS`: All available reactions users can choose on issues/prs and comments
Values can be emoji alias (:smile:) or a unicode emoji.
For custom reactions, add a tightly cropped square image to public/emoji/img/reaction_name.png
- `DEFAULT_SHOW_FULL_NAME`: **false**: Whether the full name of the users should be shown where possible. If the full name isn't set, the username will be used.
- `SEARCH_REPO_DESCRIPTION`: **true**: Whether to search within description at repository search on explore page.
- `USE_SERVICE_WORKER`: **true**: Whether to enable a Service Worker to cache frontend assets.

+ 0
- 1
docs/content/page/index.en-us.md View File

@ -274,7 +274,6 @@ Windows, on architectures like amd64, i386, ARM, PowerPC, and others.
* [DropzoneJS](http://www.dropzonejs.com/)
* [Highlight](https://highlightjs.org/)
* [Clipboard](https://zenorocha.github.io/clipboard.js/)
* [Emojify](https://github.com/Ranks/emojify.js)
* [CodeMirror](https://codemirror.net/)
* [jQuery Date Time Picker](https://github.com/xdan/datetimepicker)
* [jQuery MiniColors](https://github.com/claviska/jquery-minicolors)

+ 0
- 1
docs/content/page/index.fr-fr.md View File

@ -263,7 +263,6 @@ Le but de ce projet est de fournir de la manière la plus simple, la plus rapide
* [DropzoneJS](http://www.dropzonejs.com/)
* [Highlight](https://highlightjs.org/)
* [Clipboard](https://zenorocha.github.io/clipboard.js/)
* [Emojify](https://github.com/Ranks/emojify.js)
* [CodeMirror](https://codemirror.net/)
* [jQuery Date Time Picker](https://github.com/xdan/datetimepicker)
* [jQuery MiniColors](https://github.com/claviska/jquery-minicolors)

+ 0
- 1
docs/content/page/index.zh-cn.md View File

@ -56,7 +56,6 @@ Gitea的首要目标是创建一个极易安装,运行非常快速,安装和
* [DropzoneJS](http://www.dropzonejs.com/)
* [Highlight](https://highlightjs.org/)
* [Clipboard](https://zenorocha.github.io/clipboard.js/)
* [Emojify](https://github.com/Ranks/emojify.js)
* [CodeMirror](https://codemirror.net/)
* [jQuery Date Time Picker](https://github.com/xdan/datetimepicker)
* [jQuery MiniColors](https://github.com/claviska/jquery-minicolors)

+ 0
- 1
docs/content/page/index.zh-tw.md View File

@ -56,7 +56,6 @@ Gitea 的首要目標是建立一個容易安裝,運行快速,安装和使
* [DropzoneJS](http://www.dropzonejs.com/)
* [Highlight](https://highlightjs.org/)
* [Clipboard](https://zenorocha.github.io/clipboard.js/)
* [Emojify](https://github.com/Ranks/emojify.js)
* [CodeMirror](https://codemirror.net/)
* [jQuery Date Time Picker](https://github.com/xdan/datetimepicker)
* [jQuery MiniColors](https://github.com/claviska/jquery-minicolors)

+ 119
- 0
modules/emoji/emoji.go View File

@ -0,0 +1,119 @@
// Copyright 2020 The Gitea Authors. All rights reserved.
// Copyright 2015 Kenneth Shaw
// Use of this source code is governed by a MIT-style
// license that can be found in the LICENSE file.
package emoji
import (
"strings"
"sync"
)
// Gemoji is a set of emoji data.
type Gemoji []Emoji
// Emoji represents a single emoji and associated data.
type Emoji struct {
Emoji string
Description string
Aliases []string
UnicodeVersion string
}
var (
// codeMap provides a map of the emoji unicode code to its emoji data.
codeMap map[string]int
// aliasMap provides a map of the alias to its emoji data.
aliasMap map[string]int
// codeReplacer is the string replacer for emoji codes.
codeReplacer *strings.Replacer
// aliasReplacer is the string replacer for emoji aliases.
aliasReplacer *strings.Replacer
once sync.Once
)
func loadMap() {
once.Do(func() {
// initialize
codeMap = make(map[string]int, len(GemojiData))
aliasMap = make(map[string]int, len(GemojiData))
// process emoji codes and aliases
codePairs := make([]string, 0)
aliasPairs := make([]string, 0)
for i, e := range GemojiData {
if e.Emoji == "" || len(e.Aliases) == 0 {
continue
}
// setup codes
codeMap[e.Emoji] = i
codePairs = append(codePairs, e.Emoji, ":"+e.Aliases[0]+":")
// setup aliases
for _, a := range e.Aliases {
if a == "" {
continue
}
aliasMap[a] = i
aliasPairs = append(aliasPairs, ":"+a+":", e.Emoji)
}
}
// create replacers
codeReplacer = strings.NewReplacer(codePairs...)
aliasReplacer = strings.NewReplacer(aliasPairs...)
})
}
// FromCode retrieves the emoji data based on the provided unicode code (ie,
// "\u2618" will return the Gemoji data for "shamrock").
func FromCode(code string) *Emoji {
loadMap()
i, ok := codeMap[code]
if !ok {
return nil
}
return &GemojiData[i]
}
// FromAlias retrieves the emoji data based on the provided alias in the form
// "alias" or ":alias:" (ie, "shamrock" or ":shamrock:" will return the Gemoji
// data for "shamrock").
func FromAlias(alias string) *Emoji {
loadMap()
if strings.HasPrefix(alias, ":") && strings.HasSuffix(alias, ":") {
alias = alias[1 : len(alias)-1]
}
i, ok := aliasMap[alias]
if !ok {
return nil
}
return &GemojiData[i]
}
// ReplaceCodes replaces all emoji codes with the first corresponding emoji
// alias (in the form of ":alias:") (ie, "\u2618" will be converted to
// ":shamrock:").
func ReplaceCodes(s string) string {
loadMap()
return codeReplacer.Replace(s)
}
// ReplaceAliases replaces all aliases of the form ":alias:" with its
// corresponding unicode value.
func ReplaceAliases(s string) string {
loadMap()
return aliasReplacer.Replace(s)
}

+ 1734
- 0
modules/emoji/emoji_data.go
File diff suppressed because it is too large
View File


+ 67
- 0
modules/emoji/emoji_test.go View File

@ -0,0 +1,67 @@
// Copyright 2020 The Gitea Authors. All rights reserved.
// Copyright 2015 Kenneth Shaw
// Use of this source code is governed by a MIT-style
// license that can be found in the LICENSE file.
package emoji
import (
"reflect"
"testing"
)
func TestDumpInfo(t *testing.T) {
t.Logf("codes: %d", len(codeMap))
t.Logf("aliases: %d", len(aliasMap))
}
func TestLookup(t *testing.T) {
a := FromCode("\U0001f37a")
b := FromCode("🍺")
c := FromAlias(":beer:")
d := FromAlias("beer")
if !reflect.DeepEqual(a, b) {
t.Errorf("a and b should equal")
}
if !reflect.DeepEqual(b, c) {
t.Errorf("b and c should equal")
}
if !reflect.DeepEqual(c, d) {
t.Errorf("c and d should equal")
}
if !reflect.DeepEqual(a, d) {
t.Errorf("a and d should equal")
}
m := FromCode("\U0001f44d")
n := FromAlias(":thumbsup:")
o := FromAlias("+1")
if !reflect.DeepEqual(m, n) {
t.Errorf("m and n should equal")
}
if !reflect.DeepEqual(n, o) {
t.Errorf("n and o should equal")
}
if !reflect.DeepEqual(m, o) {
t.Errorf("m and o should equal")
}
}
func TestReplacers(t *testing.T) {
tests := []struct {
f func(string) string
v, exp string
}{
{ReplaceCodes, ":thumbsup: +1 for \U0001f37a! 🍺 \U0001f44d", ":thumbsup: +1 for :beer:! :beer: :+1:"},
{ReplaceAliases, ":thumbsup: +1 :+1: :beer:", "\U0001f44d +1 \U0001f44d \U0001f37a"},
}
for i, x := range tests {
s := x.f(x.v)
if s != x.exp {
t.Errorf("test %d `%s` expected `%s`, got: `%s`", i, x.v, x.exp, s)
}
}
}

+ 0
- 1
modules/git/command.go View File

@ -98,7 +98,6 @@ func (c *Command) RunInDirTimeoutEnvFullPipeline(env []string, timeout time.Dura
// RunInDirTimeoutEnvFullPipelineFunc executes the command in given directory with given timeout,
// it pipes stdout and stderr to given io.Writer and passes in an io.Reader as stdin. Between cmd.Start and cmd.Wait the passed in function is run.
func (c *Command) RunInDirTimeoutEnvFullPipelineFunc(env []string, timeout time.Duration, dir string, stdout, stderr io.Writer, stdin io.Reader, fn func(context.Context, context.CancelFunc) error) error {
if timeout == -1 {
timeout = DefaultCommandExecutionTimeout
}

+ 123
- 0
modules/markup/html.go View File

@ -6,6 +6,7 @@ package markup
import (
"bytes"
"fmt"
"net/url"
"path"
"path/filepath"
@ -13,6 +14,7 @@ import (
"strings"
"code.gitea.io/gitea/modules/base"
"code.gitea.io/gitea/modules/emoji"
"code.gitea.io/gitea/modules/git"
"code.gitea.io/gitea/modules/log"
"code.gitea.io/gitea/modules/markup/common"
@ -60,6 +62,13 @@ var (
// blackfriday extensions create IDs like fn:user-content-footnote
blackfridayExtRegex = regexp.MustCompile(`[^:]*:user-content-`)
// EmojiShortCodeRegex find emoji by alias like :smile:
EmojiShortCodeRegex = regexp.MustCompile(`\:[\w\+\-]+\:{1}`)
// find emoji literal: search all emoji hex range as many times as they appear as
// some emojis (skin color etc..) are just two or more chained together
emojiRegex = regexp.MustCompile(`[\x{1F000}-\x{1FFFF}|\x{2000}-\x{32ff}|\x{fe4e5}-\x{fe4ee}|\x{200D}|\x{FE0F}|\x{e0000}-\x{e007f}]+`)
)
// CSS class for action keywords (e.g. "closes: #1")
@ -154,6 +163,8 @@ var defaultProcessors = []processor{
issueIndexPatternProcessor,
sha1CurrentPatternProcessor,
emailAddressProcessor,
emojiProcessor,
emojiShortCodeProcessor,
}
type postProcessCtx struct {
@ -194,6 +205,8 @@ var commitMessageProcessors = []processor{
issueIndexPatternProcessor,
sha1CurrentPatternProcessor,
emailAddressProcessor,
emojiProcessor,
emojiShortCodeProcessor,
}
// RenderCommitMessage will use the same logic as PostProcess, but will disable
@ -226,6 +239,13 @@ var commitMessageSubjectProcessors = []processor{
mentionProcessor,
issueIndexPatternProcessor,
sha1CurrentPatternProcessor,
emojiShortCodeProcessor,
emojiProcessor,
}
var emojiProcessors = []processor{
emojiShortCodeProcessor,
emojiProcessor,
}
// RenderCommitMessageSubject will use the same logic as PostProcess and
@ -269,6 +289,17 @@ func RenderDescriptionHTML(
return ctx.postProcess(rawHTML)
}
// RenderEmoji for when we want to just process emoji and shortcodes
// in various places it isn't already run through the normal markdown procesor
func RenderEmoji(
rawHTML []byte,
) ([]byte, error) {
ctx := &postProcessCtx{
procs: emojiProcessors,
}
return ctx.postProcess(rawHTML)
}
var byteBodyTag = []byte("<body>")
var byteBodyTagClosing = []byte("</body>")
@ -319,7 +350,12 @@ func (ctx *postProcessCtx) visitNode(node *html.Node, visitText bool) {
if attr.Key == "id" && !(strings.HasPrefix(attr.Val, "user-content-") || blackfridayExtRegex.MatchString(attr.Val)) {
node.Attr[idx].Val = "user-content-" + attr.Val
}
if attr.Key == "class" && attr.Val == "emoji" {
visitText = false
}
}
// We ignore code, pre and already generated links.
switch node.Type {
case html.TextNode:
@ -406,6 +442,54 @@ func createKeyword(content string) *html.Node {
return span
}
func createEmoji(content, class, name string) *html.Node {
span := &html.Node{
Type: html.ElementNode,
Data: atom.Span.String(),
Attr: []html.Attribute{},
}
if class != "" {
span.Attr = append(span.Attr, html.Attribute{Key: "class", Val: class})
}
if name != "" {
span.Attr = append(span.Attr, html.Attribute{Key: "aria-label", Val: name})
}
text := &html.Node{
Type: html.TextNode,
Data: content,
}
span.AppendChild(text)
return span
}
func createCustomEmoji(alias, class string) *html.Node {
span := &html.Node{
Type: html.ElementNode,
Data: atom.Span.String(),
Attr: []html.Attribute{},
}
if class != "" {
span.Attr = append(span.Attr, html.Attribute{Key: "class", Val: class})
span.Attr = append(span.Attr, html.Attribute{Key: "aria-label", Val: alias})
}
img := &html.Node{
Type: html.ElementNode,
DataAtom: atom.Img,
Data: "img",
Attr: []html.Attribute{},
}
if class != "" {
img.Attr = append(img.Attr, html.Attribute{Key: "src", Val: fmt.Sprintf(`%s/img/emoji/%s.png`, setting.StaticURLPrefix, alias)})
}
span.AppendChild(img)
return span
}
func createLink(href, content, class string) *html.Node {
a := &html.Node{
Type: html.ElementNode,
@ -810,6 +894,45 @@ func fullSha1PatternProcessor(ctx *postProcessCtx, node *html.Node) {
replaceContent(node, start, end, createCodeLink(urlFull, text, "commit"))
}
// emojiShortCodeProcessor for rendering text like :smile: into emoji
func emojiShortCodeProcessor(ctx *postProcessCtx, node *html.Node) {
m := EmojiShortCodeRegex.FindStringSubmatchIndex(node.Data)
if m == nil {
return
}
alias := node.Data[m[0]:m[1]]
alias = strings.Replace(alias, ":", "", -1)
converted := emoji.FromAlias(alias)
if converted == nil {
// check if this is a custom reaction
s := strings.Join(setting.UI.Reactions, " ") + "gitea"
if strings.Contains(s, alias) {
replaceContent(node, m[0], m[1], createCustomEmoji(alias, "emoji"))
return
}
return
}
replaceContent(node, m[0], m[1], createEmoji(converted.Emoji, "emoji", converted.Description))
}
// emoji processor to match emoji and add emoji class
func emojiProcessor(ctx *postProcessCtx, node *html.Node) {
m := emojiRegex.FindStringSubmatchIndex(node.Data)
if m == nil {
return
}
codepoint := node.Data[m[0]:m[1]]
val := emoji.FromCode(codepoint)
if val != nil {
replaceContent(node, m[0], m[1], createEmoji(codepoint, "emoji", val.Description))
}
}
// sha1CurrentPatternProcessor renders SHA1 strings to corresponding links that
// are assumed to be in the same repository.
func sha1CurrentPatternProcessor(ctx *postProcessCtx, node *html.Node) {

+ 45
- 0
modules/markup/html_test.go View File

@ -8,6 +8,7 @@ import (
"strings"
"testing"
"code.gitea.io/gitea/modules/emoji"
. "code.gitea.io/gitea/modules/markup"
"code.gitea.io/gitea/modules/markup/markdown"
"code.gitea.io/gitea/modules/setting"
@ -228,6 +229,50 @@ func TestRender_email(t *testing.T) {
`<p>email@domain..com</p>`)
}
func TestRender_emoji(t *testing.T) {
setting.AppURL = AppURL
setting.AppSubURL = AppSubURL
setting.StaticURLPrefix = AppURL
test := func(input, expected string) {
expected = strings.Replace(expected, "&", "&amp;", -1)
buffer := RenderString("a.md", input, setting.AppSubURL, nil)
assert.Equal(t, strings.TrimSpace(expected), strings.TrimSpace(buffer))
}
// Make sure we can successfully match every emoji in our dataset with regex
for i := range emoji.GemojiData {
test(
emoji.GemojiData[i].Emoji,
`<p><span class="emoji" aria-label="`+emoji.GemojiData[i].Description+`">`+emoji.GemojiData[i].Emoji+`</span></p>`)
}
for i := range emoji.GemojiData {
test(
":"+emoji.GemojiData[i].Aliases[0]+":",
`<p><span class="emoji" aria-label="`+emoji.GemojiData[i].Description+`">`+emoji.GemojiData[i].Emoji+`</span></p>`)
}
//Text that should be turned into or recognized as emoji
test(
":gitea:",
`<p><span class="emoji" aria-label="gitea"><img src="`+setting.StaticURLPrefix+`/img/emoji/gitea.png"/></span></p>`)
test(
"Some text with 😄 in the middle",
`<p>Some text with <span class="emoji" aria-label="grinning face with smiling eyes">😄</span> in the middle</p>`)
test(
"Some text with :smile: in the middle",
`<p>Some text with <span class="emoji" aria-label="grinning face with smiling eyes">😄</span> in the middle</p>`)
// should match nothing
test(
"2001:0db8:85a3:0000:0000:8a2e:0370:7334",
`<p>2001:0db8:85a3:0000:0000:8a2e:0370:7334</p>`)
test(
":not exist:",
`<p>:not exist:</p>`)
}
func TestRender_ShortLinks(t *testing.T) {
setting.AppURL = AppURL
setting.AppSubURL = AppSubURL

+ 4
- 0
modules/markup/sanitizer.go View File

@ -63,6 +63,10 @@ func ReplaceSanitizer() {
// Allow unlabelled labels
sanitizer.policy.AllowNoAttrs().OnElements("label")
// Allow classes for emojis
sanitizer.policy.AllowAttrs("class").Matching(regexp.MustCompile(`emoji`)).OnElements("span")
sanitizer.policy.AllowAttrs("class").Matching(regexp.MustCompile(`emoji`)).OnElements("img")
// Allow generally safe attributes
generalSafeAttrs := []string{"abbr", "accept", "accept-charset",
"accesskey", "action", "align", "alt",

+ 27
- 0
modules/templates/helper.go View File

@ -25,6 +25,7 @@ import (
"code.gitea.io/gitea/models"
"code.gitea.io/gitea/modules/base"
"code.gitea.io/gitea/modules/emoji"
"code.gitea.io/gitea/modules/log"
"code.gitea.io/gitea/modules/markup"
"code.gitea.io/gitea/modules/repository"
@ -139,6 +140,9 @@ func NewFuncMap() []template.FuncMap {
"RenderCommitMessageLink": RenderCommitMessageLink,
"RenderCommitMessageLinkSubject": RenderCommitMessageLinkSubject,
"RenderCommitBody": RenderCommitBody,
"RenderEmoji": RenderEmoji,
"RenderEmojiPlain": emoji.ReplaceAliases,
"ReactionToEmoji": ReactionToEmoji,
"RenderNote": RenderNote,
"IsMultilineCommitMessage": IsMultilineCommitMessage,
"ThemeColorMetaTag": func() string {
@ -512,6 +516,29 @@ func RenderCommitBody(msg, urlPrefix string, metas map[string]string) template.H
return template.HTML(renderedMessage)
}
// RenderEmoji renders html text with emoji post processors
func RenderEmoji(text string) template.HTML {
renderedText, err := markup.RenderEmoji([]byte(template.HTMLEscapeString(text)))
if err != nil {
log.Error("RenderEmoji: %v", err)
return template.HTML("")
}
return template.HTML(renderedText)
}
//ReactionToEmoji renders emoji for use in reactions
func ReactionToEmoji(reaction string) template.HTML {
val := emoji.FromCode(reaction)
if val != nil {
return template.HTML(val.Emoji)
}
val = emoji.FromAlias(reaction)
if val != nil {
return template.HTML(val.Emoji)
}
return template.HTML(fmt.Sprintf(`<img src=%s/img/emoji/%s.png></img>`, setting.StaticURLPrefix, reaction))
}
// RenderNote renders the contents of a git-notes file as a commit message.
func RenderNote(msg, urlPrefix string, metas map[string]string) template.HTML {
cleanMsg := template.HTMLEscapeString(msg)

BIN
View File


+ 201
- 0
public/vendor/assets/noto-color-emoji/LICENSE View File

@ -0,0 +1,201 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

BIN
View File


+ 0
- 5
public/vendor/librejs.html View File

@ -50,11 +50,6 @@
<td><a href="https://github.com/vuejs/vue/blob/dev/LICENSE">Expat</a></td>
<td><a href="https://github.com/vuejs/vue/archive/v2.6.11.tar.gz">vue.js-v2.6.11.tar.gz</a></td>
</tr>
<tr>
<td><a href="./plugins/emojify/emojify.custom.js">emojify.custom.js</a></td>
<td><a href="http://www.freebsd.org/copyright/freebsd-license.html">Expat</a></td>
<td><a href="https://github.com/Ranks/emojify.js/archive/1.1.0.tar.gz">emojify-1.1.0.tar.gz</a></td>
</tr>
<tr>
<td><a href="../js/dropzone.js">dropzone.js</a></td>
<td><a href="https://github.com/enyo/dropzone/blob/master/LICENSE">MIT</a></td>

+ 0
- 8
public/vendor/plugins/emojify/LICENSE View File

@ -1,8 +0,0 @@
THE MIT LICENSE (MIT)
Copyright © 2014 Hassan Khan, http://hassankhan.me <contact@hassankhan.me>
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

+ 0
- 1
public/vendor/plugins/emojify/emojify.custom.js
File diff suppressed because it is too large
View File


BIN
View File


BIN
View File


BIN
View File


BIN
View File


BIN
View File


BIN
View File


BIN
View File


BIN
View File


BIN
View File


BIN
View File


BIN
View File


BIN
View File


BIN
View File


BIN
View File


BIN
View File


BIN
View File


BIN
View File


BIN
View File


BIN
View File


BIN
View File


BIN
View File


BIN
View File


BIN
View File


BIN
View File


BIN
View File


BIN
View File


BIN
View File


BIN
View File


BIN
View File


BIN
View File


BIN
View File


BIN
View File


BIN
View File


BIN
View File


BIN
View File


BIN
View File


BIN
View File


BIN
View File


BIN
View File


BIN
View File


BIN
View File


BIN
View File


BIN
View File


BIN
View File


BIN
View File


BIN
View File


BIN
View File


BIN
View File


BIN
View File


BIN
View File


BIN
View File


BIN
View File


BIN
View File


BIN
View File


BIN
View File


BIN
View File


BIN
View File


BIN
View File


BIN
View File


BIN
View File


BIN
View File


BIN
View File


BIN
View File


BIN
View File


BIN
View File


BIN
View File


BIN
View File


BIN
View File


BIN
View File


BIN
View File


BIN
View File


BIN
View File


BIN
View File


BIN
View File


BIN
View File


BIN
View File


BIN
View File


Some files were not shown because too many files changed in this diff

Loading…
Cancel
Save