Skip to content
Snippets Groups Projects
  1. Sep 14, 2014
    • Prashant Sharma's avatar
      [SPARK-3452] Maven build should skip publishing artifacts people shouldn... · f493f798
      Prashant Sharma authored
      ...'t depend on
      
      Publish local in maven term is `install`
      
      and publish otherwise is `deploy`
      
      So disabled both for following projects.
      
      Author: Prashant Sharma <prashant.s@imaginea.com>
      
      Closes #2329 from ScrapCodes/SPARK-3452/maven-skip-install and squashes the following commits:
      
      257b79a [Prashant Sharma] [SPARK-3452] Maven build should skip publishing artifacts people shouldn't depend on
      f493f798
  2. Sep 06, 2014
  3. Aug 01, 2014
    • Prashant Sharma's avatar
      SPARK-2632, SPARK-2576. Fixed by only importing what is necessary during class definition. · 14991011
      Prashant Sharma authored
      Without this patch, it imports everything available in the scope.
      
      ```scala
      
      scala> val a = 10l
      val a = 10l
      a: Long = 10
      
      scala> import a._
      import a._
      import a._
      
      scala> case class A(a: Int) // show
      case class A(a: Int) // show
      class $read extends Serializable {
        def <init>() = {
          super.<init>;
          ()
        };
        class $iwC extends Serializable {
          def <init>() = {
            super.<init>;
            ()
          };
          class $iwC extends Serializable {
            def <init>() = {
              super.<init>;
              ()
            };
            import org.apache.spark.SparkContext._;
            class $iwC extends Serializable {
              def <init>() = {
                super.<init>;
                ()
              };
              val $VAL5 = $line5.$read.INSTANCE;
              import $VAL5.$iw.$iw.$iw.$iw.a;
              class $iwC extends Serializable {
                def <init>() = {
                  super.<init>;
                  ()
                };
                import a._;
                class $iwC extends Serializable {
                  def <init>() = {
                    super.<init>;
                    ()
                  };
                  class $iwC extends Serializable {
                    def <init>() = {
                      super.<init>;
                      ()
                    };
                    case class A extends scala.Product with scala.Serializable {
                      <caseaccessor> <paramaccessor> val a: Int = _;
                      def <init>(a: Int) = {
                        super.<init>;
                        ()
                      }
                    }
                  };
                  val $iw = new $iwC.<init>
                };
                val $iw = new $iwC.<init>
              };
              val $iw = new $iwC.<init>
            };
            val $iw = new $iwC.<init>
          };
          val $iw = new $iwC.<init>
        };
        val $iw = new $iwC.<init>
      }
      object $read extends scala.AnyRef {
        def <init>() = {
          super.<init>;
          ()
        };
        val INSTANCE = new $read.<init>
      }
      defined class A
      ```
      
      With this patch, it just imports  only the necessary.
      
      ```scala
      
      scala> val a = 10l
      val a = 10l
      a: Long = 10
      
      scala> import a._
      import a._
      import a._
      
      scala> case class A(a: Int) // show
      case class A(a: Int) // show
      class $read extends Serializable {
        def <init>() = {
          super.<init>;
          ()
        };
        class $iwC extends Serializable {
          def <init>() = {
            super.<init>;
            ()
          };
          class $iwC extends Serializable {
            def <init>() = {
              super.<init>;
              ()
            };
            case class A extends scala.Product with scala.Serializable {
              <caseaccessor> <paramaccessor> val a: Int = _;
              def <init>(a: Int) = {
                super.<init>;
                ()
              }
            }
          };
          val $iw = new $iwC.<init>
        };
        val $iw = new $iwC.<init>
      }
      object $read extends scala.AnyRef {
        def <init>() = {
          super.<init>;
          ()
        };
        val INSTANCE = new $read.<init>
      }
      defined class A
      
      scala>
      
      ```
      
      This patch also adds a `:fallback` mode on being enabled it will restore the spark-shell's 1.0.0 behaviour.
      
      Author: Prashant Sharma <scrapcodes@gmail.com>
      Author: Yin Huai <huai@cse.ohio-state.edu>
      Author: Prashant Sharma <prashant.s@imaginea.com>
      
      Closes #1635 from ScrapCodes/repl-fix-necessary-imports and squashes the following commits:
      
      b1968d2 [Prashant Sharma] Added toschemaRDD to test case.
      0b712bb [Yin Huai] Add a REPL test to test importing a method.
      02ad8ff [Yin Huai] Add a REPL test for importing SQLContext.createSchemaRDD.
      ed6d0c7 [Prashant Sharma] Added a fallback mode, incase users run into issues while using repl.
      b63d3b2 [Prashant Sharma] SPARK-2632, SPARK-2576. Fixed by only importing what is necessary during class definition.
      14991011
  4. Jul 10, 2014
    • Prashant Sharma's avatar
      [SPARK-1776] Have Spark's SBT build read dependencies from Maven. · 628932b8
      Prashant Sharma authored
      Patch introduces the new way of working also retaining the existing ways of doing things.
      
      For example build instruction for yarn in maven is
      `mvn -Pyarn -PHadoop2.2 clean package -DskipTests`
      in sbt it can become
      `MAVEN_PROFILES="yarn, hadoop-2.2" sbt/sbt clean assembly`
      Also supports
      `sbt/sbt -Pyarn -Phadoop-2.2 -Dhadoop.version=2.2.0 clean assembly`
      
      Author: Prashant Sharma <prashant.s@imaginea.com>
      Author: Patrick Wendell <pwendell@gmail.com>
      
      Closes #772 from ScrapCodes/sbt-maven and squashes the following commits:
      
      a8ac951 [Prashant Sharma] Updated sbt version.
      62b09bb [Prashant Sharma] Improvements.
      fa6221d [Prashant Sharma] Excluding sql from mima
      4b8875e [Prashant Sharma] Sbt assembly no longer builds tools by default.
      72651ca [Prashant Sharma] Addresses code reivew comments.
      acab73d [Prashant Sharma] Revert "Small fix to run-examples script."
      ac4312c [Prashant Sharma] Revert "minor fix"
      6af91ac [Prashant Sharma] Ported oldDeps back. + fixes issues with prev commit.
      65cf06c [Prashant Sharma] Servelet API jars mess up with the other servlet jars on the class path.
      446768e [Prashant Sharma] minor fix
      89b9777 [Prashant Sharma] Merge conflicts
      d0a02f2 [Prashant Sharma] Bumped up pom versions, Since the build now depends on pom it is better updated there. + general cleanups.
      dccc8ac [Prashant Sharma] updated mima to check against 1.0
      a49c61b [Prashant Sharma] Fix for tools jar
      a2f5ae1 [Prashant Sharma] Fixes a bug in dependencies.
      cf88758 [Prashant Sharma] cleanup
      9439ea3 [Prashant Sharma] Small fix to run-examples script.
      96cea1f [Prashant Sharma] SPARK-1776 Have Spark's SBT build read dependencies from Maven.
      36efa62 [Patrick Wendell] Set project name in pom files and added eclipse/intellij plugins.
      4973dbd [Patrick Wendell] Example build using pom reader.
      628932b8
  5. Jun 05, 2014
  6. May 31, 2014
  7. Apr 29, 2014
    • witgo's avatar
      Improved build configuration · 030f2c21
      witgo authored
      1, Fix SPARK-1441: compile spark core error with hadoop 0.23.x
      2, Fix SPARK-1491: maven hadoop-provided profile fails to build
      3, Fix org.scala-lang: * ,org.apache.avro:* inconsistent versions dependency
      4, A modified on the sql/catalyst/pom.xml,sql/hive/pom.xml,sql/core/pom.xml (Four spaces formatted into two spaces)
      
      Author: witgo <witgo@qq.com>
      
      Closes #480 from witgo/format_pom and squashes the following commits:
      
      03f652f [witgo] review commit
      b452680 [witgo] Merge branch 'master' of https://github.com/apache/spark into format_pom
      bee920d [witgo] revert fix SPARK-1629: Spark Core missing commons-lang dependence
      7382a07 [witgo] Merge branch 'master' of https://github.com/apache/spark into format_pom
      6902c91 [witgo] fix SPARK-1629: Spark Core missing commons-lang dependence
      0da4bc3 [witgo] merge master
      d1718ed [witgo] Merge branch 'master' of https://github.com/apache/spark into format_pom
      e345919 [witgo] add avro dependency to yarn-alpha
      77fad08 [witgo] Merge branch 'master' of https://github.com/apache/spark into format_pom
      62d0862 [witgo] Fix org.scala-lang: * inconsistent versions dependency
      1a162d7 [witgo] Merge branch 'master' of https://github.com/apache/spark into format_pom
      934f24d [witgo] review commit
      cf46edc [witgo] exclude jruby
      06e7328 [witgo] Merge branch 'SparkBuild' into format_pom
      99464d2 [witgo] fix maven hadoop-provided profile fails to build
      0c6c1fc [witgo] Fix compile spark core error with hadoop 0.23.x
      6851bec [witgo] Maintain consistent SparkBuild.scala, pom.xml
      030f2c21
  8. Mar 26, 2014
    • Sean Owen's avatar
      SPARK-1325. The maven build error for Spark Tools · 1fa48d94
      Sean Owen authored
      This is just a slight variation on https://github.com/apache/spark/pull/234 and alternative suggestion for SPARK-1325. `scala-actors` is not necessary. `SparkBuild.scala` should be updated to reflect the direct dependency on `scala-reflect` and `scala-compiler`. And the `repl` build, which has the same dependencies, should also be consistent between Maven / SBT.
      
      Author: Sean Owen <sowen@cloudera.com>
      Author: witgo <witgo@qq.com>
      
      Closes #240 from srowen/SPARK-1325 and squashes the following commits:
      
      25bd7db [Sean Owen] Add necessary dependencies scala-reflect and scala-compiler to tools. Update repl dependencies, which are similar, to be consistent between Maven / SBT in this regard too.
      1fa48d94
  9. Mar 08, 2014
    • Sandy Ryza's avatar
      SPARK-1193. Fix indentation in pom.xmls · a99fb374
      Sandy Ryza authored
      Author: Sandy Ryza <sandy@cloudera.com>
      
      Closes #91 from sryza/sandy-spark-1193 and squashes the following commits:
      
      a878124 [Sandy Ryza] SPARK-1193. Fix indentation in pom.xmls
      a99fb374
  10. Mar 02, 2014
    • Patrick Wendell's avatar
      SPARK-1121: Include avro for yarn-alpha builds · c3f5e075
      Patrick Wendell authored
      This lets us explicitly include Avro based on a profile for 0.23.X
      builds. It makes me sad how convoluted it is to express this logic
      in Maven. @tgraves and @sryza curious if this works for you.
      
      I'm also considering just reverting to how it was before. The only
      real problem was that Spark advertised a dependency on Avro
      even though it only really depends transitively on Avro through
      other deps.
      
      Author: Patrick Wendell <pwendell@gmail.com>
      
      Closes #49 from pwendell/avro-build-fix and squashes the following commits:
      
      8d6ee92 [Patrick Wendell] SPARK-1121: Add avro to yarn-alpha profile
      c3f5e075
    • Patrick Wendell's avatar
      Remove remaining references to incubation · 1fd2bfd3
      Patrick Wendell authored
      This removes some loose ends not caught by the other (incubating -> tlp) patches. @markhamstra this updates the version as you mentioned earlier.
      
      Author: Patrick Wendell <pwendell@gmail.com>
      
      Closes #51 from pwendell/tlp and squashes the following commits:
      
      d553b1b [Patrick Wendell] Remove remaining references to incubation
      1fd2bfd3
  11. Feb 27, 2014
    • Sean Owen's avatar
      SPARK 1084.1 (resubmitted) · 12bbca20
      Sean Owen authored
      (Ported from https://github.com/apache/incubator-spark/pull/637 )
      
      Author: Sean Owen <sowen@cloudera.com>
      
      Closes #31 from srowen/SPARK-1084.1 and squashes the following commits:
      
      6c4a32c [Sean Owen] Suppress warnings about legitimate unchecked array creations, or change code to avoid it
      f35b833 [Sean Owen] Fix two misc javadoc problems
      254e8ef [Sean Owen] Fix one new style error introduced in scaladoc warning commit
      5b2fce2 [Sean Owen] Fix scaladoc invocation warning, and enable javac warnings properly, with plugin config updates
      007762b [Sean Owen] Remove dead scaladoc links
      b8ff8cb [Sean Owen] Replace deprecated Ant <tasks> with <target>
      12bbca20
  12. Feb 23, 2014
    • Sean Owen's avatar
      SPARK-1071: Tidy logging strategy and use of log4j · c0ef3afa
      Sean Owen authored
      Prompted by a recent thread on the mailing list, I tried and failed to see if Spark can be made independent of log4j. There are a few cases where control of the underlying logging is pretty useful, and to do that, you have to bind to a specific logger.
      
      Instead I propose some tidying that leaves Spark's use of log4j, but gets rid of warnings and should still enable downstream users to switch. The idea is to pipe everything (except log4j) through SLF4J, and have Spark use SLF4J directly when logging, and where Spark needs to output info (REPL and tests), bind from SLF4J to log4j.
      
      This leaves the same behavior in Spark. It means that downstream users who want to use something except log4j should:
      
      - Exclude dependencies on log4j, slf4j-log4j12 from Spark
      - Include dependency on log4j-over-slf4j
      - Include dependency on another logger X, and another slf4j-X
      - Recreate any log config that Spark does, that is needed, in the other logger's config
      
      That sounds about right.
      
      Here are the key changes:
      
      - Include the jcl-over-slf4j shim everywhere by depending on it in core.
      - Exclude dependencies on commons-logging from third-party libraries.
      - Include the jul-to-slf4j shim everywhere by depending on it in core.
      - Exclude slf4j-* dependencies from third-party libraries to prevent collision or warnings
      - Added missing slf4j-log4j12 binding to GraphX, Bagel module tests
      
      And minor/incidental changes:
      
      - Update to SLF4J 1.7.5, which happily matches Hadoop 2’s version and is a recommended update over 1.7.2
      - (Remove a duplicate HBase dependency declaration in SparkBuild.scala)
      - (Remove a duplicate mockito dependency declaration that was causing warnings and bugging me)
      
      Author: Sean Owen <sowen@cloudera.com>
      
      Closes #570 from srowen/SPARK-1071 and squashes the following commits:
      
      52eac9f [Sean Owen] Add slf4j-over-log4j12 dependency to core (non-test) and remove it from things that depend on core.
      77a7fa9 [Sean Owen] SPARK-1071: Tidy logging strategy and use of log4j
      c0ef3afa
  13. Feb 08, 2014
    • Mark Hamstra's avatar
      Merge pull request #542 from markhamstra/versionBump. Closes #542. · c2341c92
      Mark Hamstra authored
      Version number to 1.0.0-SNAPSHOT
      
      Since 0.9.0-incubating is done and out the door, we shouldn't be building 0.9.0-incubating-SNAPSHOT anymore.
      
      @pwendell
      
      Author: Mark Hamstra <markhamstra@gmail.com>
      
      == Merge branch commits ==
      
      commit 1b00a8a7c1a7f251b4bb3774b84b9e64758eaa71
      Author: Mark Hamstra <markhamstra@gmail.com>
      Date:   Wed Feb 5 09:30:32 2014 -0800
      
          Version number to 1.0.0-SNAPSHOT
      c2341c92
  14. Dec 15, 2013
  15. Dec 10, 2013
  16. Dec 07, 2013
  17. Oct 12, 2013
  18. Oct 06, 2013
  19. Sep 26, 2013
  20. Sep 06, 2013
  21. Sep 01, 2013
  22. Aug 21, 2013
  23. Aug 18, 2013
  24. Aug 16, 2013
  25. Aug 13, 2013
  26. Jul 16, 2013
  27. Jun 14, 2013
  28. May 01, 2013
  29. Apr 07, 2013
  30. Mar 28, 2013
  31. Mar 15, 2013
  32. Mar 01, 2013
  33. Jan 31, 2013
  34. Jan 22, 2013
Loading