SonarQube 5.6.7 无法搜索问题(ES 错误)

湿婆

我能够执行分析,尽管运行速度很慢现在我在浏览问题时面临更多问题。我们的基础设施跟不上 Elastic Search 的要求,SECCOMP 没有编译到内核中,所以我将此选项添加到 sonar.propertiessonar.search.javaAdditionalOpts=-Dbootstrap.system_call_filter=false

我还提供了大量的内存,

sonar.search.javaOpts=-Xmx2G -Xms1G -Xss1m -Djava.net.preferIPv4Stack=true \
  -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 \
  -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError

系统 64 位,16 GM RAM 和 2 个内核;运行 RHEL 6.7

并且 ES 因以下日志而崩溃。

2018.03.01 14:32:34 WARN   es[o.e.indices.breaker]  [sonar-1519909671773] [FIELDDATA] New used memory 1285367970 [1.1gb] from field [key] would be larger than configured breaker: 1278030643 [1.1gb], breaking
2018.03.01 15:09:43 WARN   es[o.e.indices.breaker]  [sonar-1519909671773] [FIELDDATA] New used memory 1285374084 [1.1gb] from field [key] would be larger than configured breaker: 1278030643 [1.1gb], breaking
2018.03.01 15:10:04 WARN   es[o.e.indices.breaker]  [sonar-1519909671773] [FIELDDATA] New used memory 1285374084 [1.1gb] from field [key] would be larger than configured breaker: 1278030643 [1.1gb], breaking
2018.03.01 15:20:15 WARN   es[o.e.indices.breaker]  [sonar-1519909671773] [FIELDDATA] New used memory 1285379720 [1.1gb] from field [key] would be larger than configured breaker: 1278030643 [1.1gb], breaking
2018.03.01 15:20:47 WARN   es[o.e.indices.breaker]  [sonar-1519909671773] [FIELDDATA] New used memory 1285379720 [1.1gb] from field [key] would be larger than configured breaker: 1278030643 [1.1gb], breaking
2018.03.01 15:21:30 WARN   es[o.e.indices.breaker]  [sonar-1519909671773] [FIELDDATA] New used memory 1285379720 [1.1gb] from field [key] would be larger than configured breaker: 1278030643 [1.1gb], breaking
2018.03.01 15:21:39 WARN   es[o.e.indices.breaker]  [sonar-1519909671773] [FIELDDATA] New used memory 1287374978 [1.1gb] from field [key] would be larger than configured breaker: 1278030643 [1.1gb], breaking
2018.03.01 15:22:02 WARN   es[o.e.indices.breaker]  [sonar-1519909671773] [FIELDDATA] New used memory 1287374978 [1.1gb] from field [key] would be larger than configured breaker: 1278030643 [1.1gb], breaking
2018.03.01 15:22:12 WARN   es[o.e.indices.breaker]  [sonar-1519909671773] [FIELDDATA] New used memory 1279785543 [1.1gb] from field [issueCreatedAt] would be larger than configured breaker: 1278030643 [1.1gb], breaking
2018.03.01 15:22:12 WARN   es[o.e.indices.breaker]  [sonar-1519909671773] [FIELDDATA] New used memory 1279798446 [1.1gb] from field [issueCreatedAt] would be larger than configured breaker: 1278030643 [1.1gb], breaking
2018.03.01 15:22:12 WARN   es[o.e.indices.breaker]  [sonar-1519909671773] [FIELDDATA] New used memory 1279787446 [1.1gb] from field [issueCreatedAt] would be larger than configured breaker: 1278030643 [1.1gb], breaking
2018.03.01 15:22:12 WARN   es[o.e.indices.breaker]  [sonar-1519909671773] [FIELDDATA] New used memory 1279771440 [1.1gb] from field [issueCreatedAt] would be larger than configured breaker: 1278030643 [1.1gb], breaking
2018.03.01 15:22:12 WARN   es[o.e.indices.breaker]  [sonar-1519909671773] [FIELDDATA] New used memory 1279800560 [1.1gb] from field [issueCreatedAt] would be larger than configured breaker: 1278030643 [1.1gb], breaking
2018.03.01 15:22:13 ERROR web[o.s.s.w.WebServiceEngine] Fail to process request http://172.17.33.86:9091/sonar/api/issues/search?p=1&ps=50&s=FILE_LINE&asc=true&additionalFields=_all&facets=types%2Cresolutions%2CcreatedAt&resolved=false&types=VULNERABILITY&sinceLeakPeriod=true&componentUuids=AWHS0koMv8wI-iczVKXw
java.lang.IllegalStateException: Fail to execute ES search request '{"from":0,"size":50,"query":{"filtered":{"query":{"match_all":{}},"filter":{"bool":{"must":[{"terms":{"modulePath":["AWHS0koMv8wI-iczVKXw"]}},{"missing":{"field":"resolution"}},{"terms":{"type":["VULNERABILITY"]}},{"range":{"issueCreatedAt":{"from":"2018-02-19T19:14:08.456Z","to":null,"include_lower":true,"include_upper":true},"_cache":false}},{"has_parent":{"query":{"filtered":{"query":{"match_all":{}},"filter":{"bool":{"must":{"or":{"filters":[{"term":{"users":"admin"}},{"term":{"groups":"Anyone"}},{"term":{"groups":"sonar-users"}},{"term":{"groups":"sonar-administrators"}}]}},"_cache":true}}}},"parent_type":"authorization"}}]}}}},"sort":[{"project":{"order":"asc","missing":"_first"}},{"filePath":{"order":"asc","missing":"_first"}},{"line":{"order":"asc","missing":"_first"}},{"severityValue":{"order":"desc","missing":"_first"}},{"key":{"order":"asc","missing":"_first"}}],"aggregations":{"types":{"global":{},"aggregations":{"types_filter":{"filter":{"bool":{"must":[{"query":{"match_all":{}}},{"terms":{"modulePath":["AWHS0koMv8wI-iczVKXw"]}},{"missing":{"field":"resolution"}},{"range":{"issueCreatedAt":{"from":"2018-02-19T19:14:08.456Z","to":null,"include_lower":true,"include_upper":true},"_cache":false}},{"has_parent":{"query":{"filtered":{"query":{"match_all":{}},"filter":{"bool":{"must":{"or":{"filters":[{"term":{"users":"admin"}},{"term":{"groups":"Anyone"}},{"term":{"groups":"sonar-users"}},{"term":{"groups":"sonar-administrators"}}]}},"_cache":true}}}},"parent_type":"authorization"}}]}},"aggregations":{"types":{"terms":{"field":"type","size":10,"min_doc_count":1,"order":{"_count":"desc"}}},"types_selected":{"terms":{"field":"type","include":"VULNERABILITY"}}}}}},"resolutions":{"global":{},"aggregations":{"resolutions_filter":{"filter":{"bool":{"must":[{"query":{"match_all":{}}},{"terms":{"modulePath":["AWHS0koMv8wI-iczVKXw"]}},{"terms":{"type":["VULNERABILITY"]}},{"range":{"issueCreatedAt":{"from":"2018-02-19T19:14:08.456Z","to":null,"include_lower":true,"include_upper":true},"_cache":false}},{"has_parent":{"query":{"filtered":{"query":{"match_all":{}},"filter":{"bool":{"must":{"or":{"filters":[{"term":{"users":"admin"}},{"term":{"groups":"Anyone"}},{"term":{"groups":"sonar-users"}},{"term":{"groups":"sonar-administrators"}}]}},"_cache":true}}}},"parent_type":"authorization"}}]}},"aggregations":{"resolutions":{"terms":{"field":"resolution","size":15,"min_doc_count":1,"order":{"_count":"desc"}}},"resolutions_missing":{"missing":{"field":"resolution"}}}}}},"createdAt":{"date_histogram":{"field":"issueCreatedAt","interval":"1d","min_doc_count":0,"pre_zone":"GMT","offset":"-3600s","format":"yyyy-MM-dd'T'HH:mm:ssZ","extended_bounds":{"min":1519067648456,"max":1519914131788}}}}}' on indices '[issues]' on types '[issue]'
    at org.sonar.server.es.request.ProxySearchRequestBuilder.get(ProxySearchRequestBuilder.java:47) ~[sonar-server-5.6.7.jar:na]
    at org.sonar.server.es.request.ProxySearchRequestBuilder.get(ProxySearchRequestBuilder.java:35) ~[sonar-server-5.6.7.jar:na]
    at org.sonar.server.issue.index.IssueIndex.search(IssueIndex.java:235) ~[sonar-server-5.6.7.jar:na]
    at org.sonar.server.issue.ws.SearchAction.doHandle(SearchAction.java:301) ~[sonar-server-5.6.7.jar:na]
    at org.sonar.server.issue.ws.SearchAction.handle(SearchAction.java:288) ~[sonar-server-5.6.7.jar:na]
    at org.sonar.server.ws.WebServiceEngine.execute(WebServiceEngine.java:107) ~[sonar-server-5.6.7.jar:na]
    at sun.reflect.GeneratedMethodAccessor228.invoke(Unknown Source) ~[na:na]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_91]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_91]
    at org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:425) [jruby-complete-1.7.9.jar:na]
    at org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:292) [jruby-complete-1.7.9.jar:na]
    at org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:44) [jruby-complete-1.7.9.jar:na]

是的,它在崩溃之前确实恳求了一段时间,并拒绝在 UI 上显示任何问题。请帮助,所有设置升级到 SQ 5.6.7 以便稍后我们可以升级到 SQ 6.7.1 但现在我们仍在生产中运行 SQ 4.5.4(它能够处理如此大量的数据)

我们的数据库有多大?我们的数据库拥有接近 2700 万个 LOC,数据库上有 560 万个关闭/打开的问题。

湿婆

正确的解决方案

  • 升级到 SonarQube 6.1 LTS
  • 如果您是 Oracle DB,请确保分析表。即,保持索引是最新的
  • 如果您没有升级的空间并且索引是最新的,但您遇到缓慢,请为 Database Cleaner 设置最少的周数,甚至比 SonarQube 推荐的更少,然后运行分析。

当 SonarQube 删除旧快照时,所有相关的旧问题都将被正确删除,只有在运行新分析时才会在组件上执行 dbcleaner。如果您有幽灵项目(不再在 SCM 中),请创建空项目并使用相同的密钥清除数据库。

愚蠢的黑客

长期以来,我尝试按照问题SONAR-8067(在 Ann 的评论中链接)中的建议更改 SonarQube 的代码但是我做不到,描述太高级了,我几乎没有时间疯狂地进行逆向工程,所以我选择了蛮力。

特别提及:我相信我们已经对封闭问题制定了非常严厉的政策,但他们仍然很顽固。关闭时删除问题

由于扫描缩小的 JS,我们的数据库每天都在呈指数级增长,现在它在 Issues 表中有 680 万行,我别无选择地执行了以下命令。

delete from issues where status = 'CLOSED' or status = 'RESOLVED'

这条sql语句耗时14分钟,从表中清除了450万行。后来从 SonarQube 主目录中删除datatemp目录并重新启动 SonarQube,现在 ES 正在呼吸。但故事还没有结束。现在仪表板不稳定,680 万行又回到了表中!(我不知道这是怎么发生的,我怀疑 SonarQube 中有一些流氓代码)。

Project 主页仪表板中未显示的问题

问题仍然可以从问题、度量和代码仪表板浏览。像我们一样,如果您有自定义仪表板添加问题小部件,

尽管问题显示在仪表板 > 仪表板:问题小部件中

版主注意:有时愚蠢的黑客可能就足够了。

本文收集自互联网,转载请注明来源。

如有侵权,请联系 [email protected] 删除。

编辑于
0

我来说两句

0 条评论
登录 后参与评论

相关文章

PhoneGap resolveLocalFileSystemURL问题(错误5)

Xcode 6故事板屏幕尺寸在iPhone5s(iOS 7)中错误,但在(iOS 8)中不是

正则表达式可检测IE 5、6、7和8,但无法检测其他

HTTP错误403.14-禁止(ASP.NET MVC 6,ASP.NET 5,IIS 7,Windows Server 2008)

ASP .NET 5 MVC 6 beta 7升级:“具有等效标识的多个程序集”问题

eslint解析错误:ecmaVersion必须为3、5、6或7

如何解决Android Studio Canary 5、6和7上的即时运行问题

SonarQube-无法编写消息错误

Angular 5&6测试错误:无法加载指令AppComponent的摘要

角度从5错误更新为6错误

PromiseKit 6错误无法转换错误

如何创建Angular 5/6/7逆时针倒数(毫秒)?

角度5到7更新后无法生成产品,错误

从Angular 6迁移到Angular 7会导致错误-无法解决'core-js / es7 / reflect'

Errno 14 PYCURL错误6; 无法解析Cloudera Manager 7.x升级中的主机

SonarQube 8.5.1-无法启动服务-包装器错误

XCode 5和iOS 6和7

Ruby排序-为什么“预期:[7、6、5、5、4、3、3]”时rspec错误似乎与“获得:[7、6、5、5、4、3、3]”相同?

为什么打印5 5 5而不是5 6 7?(多个单例呼叫)

在用户键入(1 2 3 4 5 6 7 8 9 10)时无法显示正确的最大值

Sonarqube安装错误

安装Windows 7时,出现CDBOOT错误5:无法从CD引导;无法从CD引导。为什么?

ParseFacebookUtilsV4启用位码错误Xcode 7 Beta 6

在哪里可以找到es5,es6和es7功能的完整列表?

将 es7 转换为 es6 错误意外标识符

编译到 ES6 工作但失败,错误 TS2693 与 ES5

SonarQube 中的 FileInputStream 错误

Angular 6 的 sonarqube 和 azure devops 的代码覆盖问题

为什么 int arr[]={0,3,2,4,5,6,7}; 给出错误和 int arr[7]={0,3,2,4,5,6,7}; 不是