You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

214 lines
6.4 KiB

Add a storage layer for attachments (#11387) * Add a storage layer for attachments * Fix some bug * fix test * Fix copyright head and lint * Fix bug * Add setting for minio and flags for migrate-storage * Add documents * fix lint * Add test for minio store type on attachments * fix test * fix test * Apply suggestions from code review Co-authored-by: guillep2k <18600385+guillep2k@users.noreply.github.com> * Add warning when storage migrated successfully * Fix drone * fix test * rebase * Fix test * display the error on console * Move minio test to amd64 since minio docker don't support arm64 * refactor the codes * add trace * Fix test * remove log on xorm * Fi download bug * Add a storage layer for attachments * Add setting for minio and flags for migrate-storage * fix lint * Add test for minio store type on attachments * Apply suggestions from code review Co-authored-by: guillep2k <18600385+guillep2k@users.noreply.github.com> * Fix drone * fix test * Fix test * display the error on console * Move minio test to amd64 since minio docker don't support arm64 * refactor the codes * add trace * Fix test * Add URL function to serve attachments directly from S3/Minio * Add ability to enable/disable redirection in attachment configuration * Fix typo * Add a storage layer for attachments * Add setting for minio and flags for migrate-storage * fix lint * Add test for minio store type on attachments * Apply suggestions from code review Co-authored-by: guillep2k <18600385+guillep2k@users.noreply.github.com> * Fix drone * fix test * Fix test * display the error on console * Move minio test to amd64 since minio docker don't support arm64 * don't change unrelated files * Fix lint * Fix build * update go.mod and go.sum * Use github.com/minio/minio-go/v6 * Remove unused function * Upgrade minio to v7 and some other improvements * fix lint * Fix go mod Co-authored-by: guillep2k <18600385+guillep2k@users.noreply.github.com> Co-authored-by: Tyler <tystuyfzand@gmail.com>
3 years ago
Add a storage layer for attachments (#11387) * Add a storage layer for attachments * Fix some bug * fix test * Fix copyright head and lint * Fix bug * Add setting for minio and flags for migrate-storage * Add documents * fix lint * Add test for minio store type on attachments * fix test * fix test * Apply suggestions from code review Co-authored-by: guillep2k <18600385+guillep2k@users.noreply.github.com> * Add warning when storage migrated successfully * Fix drone * fix test * rebase * Fix test * display the error on console * Move minio test to amd64 since minio docker don't support arm64 * refactor the codes * add trace * Fix test * remove log on xorm * Fi download bug * Add a storage layer for attachments * Add setting for minio and flags for migrate-storage * fix lint * Add test for minio store type on attachments * Apply suggestions from code review Co-authored-by: guillep2k <18600385+guillep2k@users.noreply.github.com> * Fix drone * fix test * Fix test * display the error on console * Move minio test to amd64 since minio docker don't support arm64 * refactor the codes * add trace * Fix test * Add URL function to serve attachments directly from S3/Minio * Add ability to enable/disable redirection in attachment configuration * Fix typo * Add a storage layer for attachments * Add setting for minio and flags for migrate-storage * fix lint * Add test for minio store type on attachments * Apply suggestions from code review Co-authored-by: guillep2k <18600385+guillep2k@users.noreply.github.com> * Fix drone * fix test * Fix test * display the error on console * Move minio test to amd64 since minio docker don't support arm64 * don't change unrelated files * Fix lint * Fix build * update go.mod and go.sum * Use github.com/minio/minio-go/v6 * Remove unused function * Upgrade minio to v7 and some other improvements * fix lint * Fix go mod Co-authored-by: guillep2k <18600385+guillep2k@users.noreply.github.com> Co-authored-by: Tyler <tystuyfzand@gmail.com>
3 years ago
Add a storage layer for attachments (#11387) * Add a storage layer for attachments * Fix some bug * fix test * Fix copyright head and lint * Fix bug * Add setting for minio and flags for migrate-storage * Add documents * fix lint * Add test for minio store type on attachments * fix test * fix test * Apply suggestions from code review Co-authored-by: guillep2k <18600385+guillep2k@users.noreply.github.com> * Add warning when storage migrated successfully * Fix drone * fix test * rebase * Fix test * display the error on console * Move minio test to amd64 since minio docker don't support arm64 * refactor the codes * add trace * Fix test * remove log on xorm * Fi download bug * Add a storage layer for attachments * Add setting for minio and flags for migrate-storage * fix lint * Add test for minio store type on attachments * Apply suggestions from code review Co-authored-by: guillep2k <18600385+guillep2k@users.noreply.github.com> * Fix drone * fix test * Fix test * display the error on console * Move minio test to amd64 since minio docker don't support arm64 * refactor the codes * add trace * Fix test * Add URL function to serve attachments directly from S3/Minio * Add ability to enable/disable redirection in attachment configuration * Fix typo * Add a storage layer for attachments * Add setting for minio and flags for migrate-storage * fix lint * Add test for minio store type on attachments * Apply suggestions from code review Co-authored-by: guillep2k <18600385+guillep2k@users.noreply.github.com> * Fix drone * fix test * Fix test * display the error on console * Move minio test to amd64 since minio docker don't support arm64 * don't change unrelated files * Fix lint * Fix build * update go.mod and go.sum * Use github.com/minio/minio-go/v6 * Remove unused function * Upgrade minio to v7 and some other improvements * fix lint * Fix go mod Co-authored-by: guillep2k <18600385+guillep2k@users.noreply.github.com> Co-authored-by: Tyler <tystuyfzand@gmail.com>
3 years ago
  1. // Copyright 2016 The Gitea Authors. All rights reserved.
  2. // Use of this source code is governed by a MIT-style
  3. // license that can be found in the LICENSE file.
  4. package models
  5. import (
  6. "fmt"
  7. "io/ioutil"
  8. "math"
  9. "net/url"
  10. "os"
  11. "path/filepath"
  12. "testing"
  13. "code.gitea.io/gitea/modules/base"
  14. "code.gitea.io/gitea/modules/setting"
  15. "code.gitea.io/gitea/modules/storage"
  16. "code.gitea.io/gitea/modules/util"
  17. "github.com/stretchr/testify/assert"
  18. "github.com/unknwon/com"
  19. "xorm.io/xorm"
  20. "xorm.io/xorm/names"
  21. )
  22. // NonexistentID an ID that will never exist
  23. const NonexistentID = int64(math.MaxInt64)
  24. // giteaRoot a path to the gitea root
  25. var (
  26. giteaRoot string
  27. fixturesDir string
  28. )
  29. func fatalTestError(fmtStr string, args ...interface{}) {
  30. fmt.Fprintf(os.Stderr, fmtStr, args...)
  31. os.Exit(1)
  32. }
  33. // MainTest a reusable TestMain(..) function for unit tests that need to use a
  34. // test database. Creates the test database, and sets necessary settings.
  35. func MainTest(m *testing.M, pathToGiteaRoot string) {
  36. var err error
  37. giteaRoot = pathToGiteaRoot
  38. fixturesDir = filepath.Join(pathToGiteaRoot, "models", "fixtures")
  39. if err = CreateTestEngine(fixturesDir); err != nil {
  40. fatalTestError("Error creating test engine: %v\n", err)
  41. }
  42. setting.AppURL = "https://try.gitea.io/"
  43. setting.RunUser = "runuser"
  44. setting.SSH.Port = 3000
  45. setting.SSH.Domain = "try.gitea.io"
  46. setting.Database.UseSQLite3 = true
  47. setting.RepoRootPath, err = ioutil.TempDir(os.TempDir(), "repos")
  48. if err != nil {
  49. fatalTestError("TempDir: %v\n", err)
  50. }
  51. setting.AppDataPath, err = ioutil.TempDir(os.TempDir(), "appdata")
  52. if err != nil {
  53. fatalTestError("TempDir: %v\n", err)
  54. }
  55. setting.AppWorkPath = pathToGiteaRoot
  56. setting.StaticRootPath = pathToGiteaRoot
  57. setting.GravatarSourceURL, err = url.Parse("https://secure.gravatar.com/avatar/")
  58. if err != nil {
  59. fatalTestError("url.Parse: %v\n", err)
  60. }
  61. setting.Attachment.Path = filepath.Join(setting.AppDataPath, "attachments")
  62. setting.LFS.ContentPath = filepath.Join(setting.AppDataPath, "lfs")
  63. if err = storage.Init(); err != nil {
  64. fatalTestError("storage.Init: %v\n", err)
  65. }
  66. if err = util.RemoveAll(setting.RepoRootPath); err != nil {
  67. fatalTestError("util.RemoveAll: %v\n", err)
  68. }
  69. if err = com.CopyDir(filepath.Join(pathToGiteaRoot, "integrations", "gitea-repositories-meta"), setting.RepoRootPath); err != nil {
  70. fatalTestError("com.CopyDir: %v\n", err)
  71. }
  72. exitStatus := m.Run()
  73. if err = util.RemoveAll(setting.RepoRootPath); err != nil {
  74. fatalTestError("util.RemoveAll: %v\n", err)
  75. }
  76. if err = util.RemoveAll(setting.AppDataPath); err != nil {
  77. fatalTestError("util.RemoveAll: %v\n", err)
  78. }
  79. os.Exit(exitStatus)
  80. }
  81. // CreateTestEngine creates a memory database and loads the fixture data from fixturesDir
  82. func CreateTestEngine(fixturesDir string) error {
  83. var err error
  84. x, err = xorm.NewEngine("sqlite3", "file::memory:?cache=shared&_txlock=immediate")
  85. if err != nil {
  86. return err
  87. }
  88. x.SetMapper(names.GonicMapper{})
  89. if err = x.StoreEngine("InnoDB").Sync2(tables...); err != nil {
  90. return err
  91. }
  92. switch os.Getenv("GITEA_UNIT_TESTS_VERBOSE") {
  93. case "true", "1":
  94. x.ShowSQL(true)
  95. }
  96. return InitFixtures(fixturesDir)
  97. }
  98. // PrepareTestDatabase load test fixtures into test database
  99. func PrepareTestDatabase() error {
  100. return LoadFixtures()
  101. }
  102. // PrepareTestEnv prepares the environment for unit tests. Can only be called
  103. // by tests that use the above MainTest(..) function.
  104. func PrepareTestEnv(t testing.TB) {
  105. assert.NoError(t, PrepareTestDatabase())
  106. assert.NoError(t, util.RemoveAll(setting.RepoRootPath))
  107. metaPath := filepath.Join(giteaRoot, "integrations", "gitea-repositories-meta")
  108. assert.NoError(t, com.CopyDir(metaPath, setting.RepoRootPath))
  109. base.SetupGiteaRoot() // Makes sure GITEA_ROOT is set
  110. }
  111. type testCond struct {
  112. query interface{}
  113. args []interface{}
  114. }
  115. // Cond create a condition with arguments for a test
  116. func Cond(query interface{}, args ...interface{}) interface{} {
  117. return &testCond{query: query, args: args}
  118. }
  119. func whereConditions(sess *xorm.Session, conditions []interface{}) {
  120. for _, condition := range conditions {
  121. switch cond := condition.(type) {
  122. case *testCond:
  123. sess.Where(cond.query, cond.args...)
  124. default:
  125. sess.Where(cond)
  126. }
  127. }
  128. }
  129. func loadBeanIfExists(bean interface{}, conditions ...interface{}) (bool, error) {
  130. sess := x.NewSession()
  131. defer sess.Close()
  132. whereConditions(sess, conditions)
  133. return sess.Get(bean)
  134. }
  135. // BeanExists for testing, check if a bean exists
  136. func BeanExists(t testing.TB, bean interface{}, conditions ...interface{}) bool {
  137. exists, err := loadBeanIfExists(bean, conditions...)
  138. assert.NoError(t, err)
  139. return exists
  140. }
  141. // AssertExistsAndLoadBean assert that a bean exists and load it from the test
  142. // database
  143. func AssertExistsAndLoadBean(t testing.TB, bean interface{}, conditions ...interface{}) interface{} {
  144. exists, err := loadBeanIfExists(bean, conditions...)
  145. assert.NoError(t, err)
  146. assert.True(t, exists,
  147. "Expected to find %+v (of type %T, with conditions %+v), but did not",
  148. bean, bean, conditions)
  149. return bean
  150. }
  151. // GetCount get the count of a bean
  152. func GetCount(t testing.TB, bean interface{}, conditions ...interface{}) int {
  153. sess := x.NewSession()
  154. defer sess.Close()
  155. whereConditions(sess, conditions)
  156. count, err := sess.Count(bean)
  157. assert.NoError(t, err)
  158. return int(count)
  159. }
  160. // AssertNotExistsBean assert that a bean does not exist in the test database
  161. func AssertNotExistsBean(t testing.TB, bean interface{}, conditions ...interface{}) {
  162. exists, err := loadBeanIfExists(bean, conditions...)
  163. assert.NoError(t, err)
  164. assert.False(t, exists)
  165. }
  166. // AssertExistsIf asserts that a bean exists or does not exist, depending on
  167. // what is expected.
  168. func AssertExistsIf(t *testing.T, expected bool, bean interface{}, conditions ...interface{}) {
  169. exists, err := loadBeanIfExists(bean, conditions...)
  170. assert.NoError(t, err)
  171. assert.Equal(t, expected, exists)
  172. }
  173. // AssertSuccessfulInsert assert that beans is successfully inserted
  174. func AssertSuccessfulInsert(t testing.TB, beans ...interface{}) {
  175. _, err := x.Insert(beans...)
  176. assert.NoError(t, err)
  177. }
  178. // AssertCount assert the count of a bean
  179. func AssertCount(t testing.TB, bean interface{}, expected interface{}) {
  180. assert.EqualValues(t, expected, GetCount(t, bean))
  181. }
  182. // AssertInt64InRange assert value is in range [low, high]
  183. func AssertInt64InRange(t testing.TB, low, high, value int64) {
  184. assert.True(t, value >= low && value <= high,
  185. "Expected value in range [%d, %d], found %d", low, high, value)
  186. }